acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Code Localization through Repair-Context Completeness: From Structural and Code Coverage Analysis to Context Augmentation

Abstract

Repository-level code localization is difficult because a single repair may span multiple distant source regions. Existing evaluations often report relevant hits or file coverage, but these metrics alone are insufficient to determine whether a bounded repair context contains all regions involved in a fix. We characterize this gap through repair-context completeness, defined as containment of the full *repair closure*, which consists of the pre-fix regions changed by a reference patch. Within this scope, we examine the assumption that directly using execution coverage improves line-level localization. Experiments across six dimensions show that execution evidence can expose more candidates without reliably assembling complete repair contexts under the tested policies. Accordingly, we introduce Complete-Fix Reachability–Realization (CFRR), a measurement framework that attributes incomplete contexts to failures at distinct stages of localization and context construction. Guided by this diagnosis, we propose TraceFill, which preserves the semantic localization base and uses repeat-consistent execution coverage to fill remaining context capacity. Our evaluation spans three models and starts with 1,100 instances from three datasets. TraceFill increases full repair-closure containment from 33.38% to 41.24% compared with the semantic baseline, a gain of 7.86 percentage points. In the downstream evaluation, repair success increases over Native for all three repair generators, with absolute gains of 0.60–1.69 percentage points and relative gains of 23.5–32.7%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
code localization, automated program repair, large language models, execution coverage, repair-context completeness, SWE-bench

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JS18OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates code localization from an interesting perspective of repair-context completeness. It proposes CFRR to analyze why contexts are incomplete and proposes TraceFill to supplement existing localization context with code regions derived during execution. Experiments show that using locations identified by TraceFill can help to increase the effectiveness of repair approaches.

Critical strengths

The proposed research problem is interesting and worth studying. Identifying a relevant file or suspicious line does not necessarily give a repair model enough context to fix a bug. Thus, looking at localization through the lens of repair completeness is useful and novel to the best of my knowledge

Critical weaknesses

First, the presentation is quite difficult for readers to follow. Specifically, I found it difficult to follow how the six exploratory studies lead to CFRR and then to TraceFill. Many details and justifications are missing, so I cannot properly evaluate the soundness of these experiments. For example, what are the overarching objectives of the six exploratory studies? What are the implications of these exploratory results and how do they motivate CFRR and TraceFill? For CFRR, why are these four criteria chosen, and what distinct insight does each provide? For TraceFill, why should exception sites receive higher priority than call/return endpoints and other executed lines? These are just a few questions that I repeatedly ask myself as a reader and cannot find sufficient answers in the current submission.

Second, the contribution beyond existing work is insufficiently explained. CFRR separates failures in 4 sub-criteria: exposure, reachability, capacity and realization. Among these criteria: exposure and reachability are not new and widely-used by prior works despite the new name provided by the papers. Capacity and realization are somewhat new. However, Table 2 reports no capacity failures, so I am not convinced that these criteria are really important, especially given the long-context capabilities of frontier models. Meanwhile, the need for realization is also insufficiently motivated. My understanding is that it measures whether repair tools really select correct context from candidates provided by code localization approaches. Thus, its outcome depends on the repair tool effectiveness, so using it to evaluate code localization will introduce additional confounding factors. Besides, the novelty of TraceFill also remains unclear to me. Many prior works such as AutoCodeRover[1] have used execution information to improve code localization and demonstrate its effectiveness. TraceFill differs in preserving the initial context and greedily adding source windows under a residual budget. However, the paper does not sufficiently explain why these particular choices constitute a substantive advance or compare them with closely related alternatives under the same budget.

Finally, many experimental designs are questionable and yield threats to the validity of the findings. Specifically, CFRR is evaluated on a set of 117 single-file bugs whose reference patches lie on the crash path. Unfortunately, for single-file bugs, some criteria of CFRR is not fully evaluated, e.g., exposure reduces to checking whether one target file is retrieved, rather than whether all required files are available. Besides, the repair improvements may largely come from problems in the Native baseline. Native provides no source code for 798 of 1,004 bugs, often due to parsing failures, and all net repair gains for DeepSeek and GPT occur in these empty-context cases (Section 5.3 and Table 5b). The authors should fix the parsing and configuration problems and evaluate whether the improvements remain. The comparisons also do not establish the effectiveness of TraceFill’s selection strategy. TraceFill adds up to 414 source lines, but the controls restrict candidate selection, while the ablations provide limited support for its design choices. Comparisons with expanded Native windows, enclosing functions, and standard fault-localization rankings under the same budget would better establish its effectiveness.

[1] Zhang, Yuntong, et al. "Autocoderover: Autonomous program improvement." Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024.

Questions for authors

Q1. What is the overarching objective of the six exploratory studies, and how do their findings motivate CFRR’s four criteria and TraceFill’s design?

Q2. What is the novelty of CFRR criteria provided beyond existing localization metrics?

Q3. What is TraceFill’s main methodological contribution beyond existing execution-based approaches such as AutoCodeRover? Why should preserving the initial context and greedily adding execution-derived windows provide a substantive advantage?

Q4. Why is CFRR evaluated only on 117 single-file bugs whose reference patches lie on the crash path? Would including multi-file and off-path bugs change the findings about the four criteria?

Q5. Do TraceFill’s improvements remain after correcting Native’s parsing and configuration problems, and how does it compare with expanded Native windows, enclosing functions, and standard fault-localization rankings under the same context budget?

Reviewer_dERQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits code localization through repair-context completeness, defined as containing all reference-mapped edit locations within a source-context budget. It introduces CFRR to distinguish failures in exposure, reachability, capacity, and realization, and proposes TRACEFILL, which preserves existing localization context while adding execution-derived source windows. Across three models and 763 eligible defects, TRACEFILL improves complete containment by 7.86 percentage points at the largest budget. Additional repair experiments report smaller absolute improvements, although these results are substantially affected by empty baseline contexts and localization-interface failures.

Critical strengths
  1. The paper is well motivated: retrieving a relevant file does not necessarily provide all locations involved in a repair. The staged analysis offers a useful distinction between candidate availability and context selection.

  2. The evaluation includes multiple models, budget settings, matched controls, and uncertainty estimates. The explicit reporting of interface failures, repair regressions, and reference-relative limitations is particularly valuable.

Critical weaknesses
  1. The downstream evidence is weakened by a malfunctioning baseline. Appendix J.3 reports that 79.48% of Native contexts are empty. Among 581 first-stage parsing failures, 567 responses contain valid repository paths rejected because of formatting. Most repair gains occur in the empty-context stratum; with nonempty Native contexts, DeepSeek and GPT show no net improvement, while Qwen gains four repairs. Although transparently disclosed, these results primarily establish recovery from missing context, rather than improvement over a functioning localization system. A corrected-parser evaluation, with Native localization available independently of trace admission, is necessary to assess practical value.

  2. The relationship between completeness and useful repair context remains insufficiently established. The metric measures containment of reference edit sites, not semantic sufficiency: necessary supporting information may lie elsewhere, and alternative correct patches may edit different locations. The paper acknowledges this distinction, but the compromised repair comparison leaves limited evidence that optimizing this metric improves actual repair. Moreover, containment gains are substantially smaller for cross-file defects than single-file defects (1.29 versus 9.54 percentage points), limiting the evidence for the broader completeness motivation.

  3. The comparisons do not fully establish TRACEFILL's advantage over practical alternatives. The matched containment controls exclude stable-trace anchors, and no standard spectrum-based ranking or matched-extra-context repair comparison is provided. Furthermore, single-trace and intersection variants achieve identical containment on the admitted cohort, while end-to-end execution cost is unreported. The benefit of the particular selection policy and two-run requirement therefore remains unclear.

Questions for authors

Does improved completeness predict repair success beyond file coverage and context size, particularly for cross-file defects?

How does TRACEFILL compare with spectrum-based ranking and single-trace augmentation under matched context budgets and measured end-to-end cost?

Reviewer_UMQhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is very difficult to read; what follows is my best-effort understanding. The paper studies whether the code context given to an LLM for repository-level repair (SWE-bench) contains all lines changed by the reference patch. It reports that an existing LLM-based localizer (“Native”) often produces incomplete contexts, and that several direct uses of execution signals do not fix this. It then proposes TraceFill, which keeps Native’s context and adds code regions executed by the failing test, up to an additional line budget. Experiments report higher containment of reference-patch lines (33.4% → 41.2%) and small downstream gains in resolve rate (0.6–1.7 points).

Critical strengths

The paper attempts a thorough investigation of context construction for code repair and is transparent about its limitations and artifacts.

Critical weaknesses
  1. Clarity. The paper introduces dense terminology, such as “repair closure,” “candidate reachability,” “semantic localization base,” “Native,” “KEEP,” and COMPLETE@3. These terms are either defined only formally or not explained in plain terms, and they are not related to standard terms in prior work. Section 3 refers to RQ1-RQ4, but the questions are never explicitly stated, and the progression from Section 3 to Section 5 is hard to follow. Results also span many differently sized "cohorts". Overall, I could not confidently identify the paper’s motivation, main claims, and findings.
  2. Experimental setup and baseline. The benchmarks and models appear only in Section 5. The main text never explains what Native is or where the development "cohorts" in Sections 3-4 come from. Inference settings appear only in Appendix N.2 and include a 300-token output limit for localization. With this limit, 74% of DeepSeek’s Native contexts are empty (Table 42), as are 79% of the repair-stage contexts. The authors report that most of the downstream gain comes from these empty cases, so the improvement may largely reflect a weak baseline.
  3. Possible test leakage. TraceFill and the RQ2 experiments execute the benchmark’s FAIL_TO_PASS tests (Figure 4; Appendix K.2), including an expected-failure check on pre-fix code. In SWE-bench, these tests come from the test patch accompanying the reference fix and are hidden during standard evaluation. If so, TraceFill uses information unavailable at inference time, and its comparisons with Native and the controls are not on equal footing. The paper does not discuss this.
Questions for authors
  1. Can you define the new terminology in plain terms, relate it to existing work, and explain why new terms are needed?
  2. With that clarification, can you restate the main hypotheses, experiments, and findings?
  3. What exactly are the Native localizer, the development cohorts, and the inference settings? How do the results change with a working parser and a realistic token limit?
  4. Does TraceFill execute the hidden FAIL_TO_PASS tests?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.