acceptodds
Under review as a conference paper at ICLR 2027

What File-Level Hits Hide in Bug Localization: Paths, Functions, and Repair

Abstract

Bug localization narrows an issue report to code that needs inspection or repair. File recall measures whether the right file is found, but does not distinguish useful path cues from code evidence or establish that the edit site has been located. Controlled experiments with CodeGRIP reveal both gaps. Changing displayed paths while holding reports, code previews and candidate pools fixed, we find that readable names from unrelated repositories can be more disruptive than opaque identifiers even when code remains visible. On issue-held-out SWE-bench Lite, this PathAlias intervention lowers Recall@1 by 21.8 percentage points for a code-in-prompt Qwen2.5-14B recipe without path augmentation and 4.9 points with it. Separate matched code-preview controls reveal sharply different opaque-path performance behind similar original-path scores. On GREPO, following file predictions through function selection, about one quarter of file-hit records across four rerankers also yield a changed-function hit under their own scorers. Under either tested shared scorer, neither code-visible reranker has a reliable advantage over path-only on common file wins. In separate Lite repair populations, correct supplied functions are associated with higher resolution among non-empty patches, whereas file-given comparisons remain inconclusive. File recall therefore needs to be read alongside path interventions and function outcomes, with repair interpreted under the location supplied to the fixer.

Primary area:
datasets and benchmarks
Keywords:
Bug localization, Path sensitivity, Code reranking, Function localization, Program repair

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JdhVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes a set of diagnostic evaluations for interpreting bug localization results. It argues that file-level recall can obscure two important limitations: a reranker may rely heavily on file paths even when code is available, and identifying the correct file does not necessarily identify the function that needs to change. Using fixed candidate pools, the authors test rerankers with readable aliases drawn from unrelated repositories, opaque path identifiers, path exchanges, and historical file names. On SWE-bench Lite, the PathAlias intervention reduces Recall@1 by 21.8 percentage points for a Qwen2.5-14B reranker trained without path augmentation, compared with 4.9 points for a path-augmented version. On GREPO, only 585 of 2,356 file-hit records (24.8%) also identify a changed function using each system’s own function scorer. Finally, separate repair experiments show that supplying the correct function is strongly associated with successful resolution when the fixer is instructed to rewrite that function, while the evidence from file-given repair remains inconclusive.

Critical strengths
  1. It reveals distinct underlying behaviors associated with identical scores. Under the original path setting, the Recall@1 scores for the two 14B models—one with code and one without—were comparable (57.0% vs. 55.7%); however, when the paths were replaced with unreadable identifiers, the figures shifted to 39.7% and 0.3%, respectively. This compellingly demonstrates that standard file-level scores are insufficient to determine whether a model possesses the capability to utilize code.

  2. The paper analyzes the complete "file-to-function" workflow, compares results using a shared function-level scorer, and examines two types of repair inputs: "file-provided" and "function-provided." Furthermore, the authors avoid interpreting confidence intervals that span zero as evidence of equivalence between methods, nor do they mistake correlations observed in the repair experiments for causal effects.

Critical weaknesses
  1. There is a lack of experiments on state-of-the-art LLMs; I believe the issues raised by the authors do not pose significant obstacles for these advanced models.

  2. I feel this evaluation method somewhat deviates from the core objective of the task. Placing too much emphasis on the effects of path intervention shifts the focus away from the benchmark's primary evaluation criteria.

  3. The shared function scorer utilizes file paths, summaries, and function names but does not read the function bodies; furthermore, comparisons are restricted to instances where both file rankers correctly identify the target file. Consequently, this does not rule out the possibility that code-assisted file ranking could yield downstream benefits when integrated with more powerful function locators or a complete workflow.

Questions for authors
  1. Please conduct experiments using additional state-of-the-art models.

  2. Please explain the significance of this evaluation for the task at hand.

  3. Please revise the wording of the paper as much as possible; I had to read it twice to understand the task involved, and I do not believe other readers will have the patience to do the same. Additionally, the figures in the paper look overly "AI-generated."

Reviewer_LpMzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • Studies whether file-level recall hides path dependence and incomplete function localization.
  • Uses controlled path interventions, matched code-preview controls, function-level checks, and separate repair analyses.
Critical strengths
  • Controlled path interventions separate displayed-path effects from code evidence.
  • The paper identifies an important gap between file hits, function localization, and repair outcomes.
Critical weaknesses
  • The experiments focus on pointwise rerankers and fixed candidate pools, limiting scope.
  • The separate repair analyses do not establish a strong causal link to end-to-end repair.
Questions for authors
  • Can the main findings be reproduced for listwise localization or larger candidate pools?
  • Which function-level metric best predicts repair success?
Reviewer_imVGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what file-level retrieval accuracy fails to capture in bug localization. Using CodeGRIP as an experimental testbed, it introduces controlled path interventions that change displayed filenames while holding issue reports, code previews, and candidate pools fixed. The results show that similar original-path scores can conceal substantially different robustness to misleading paths and different use of code information. The paper also follows file predictions through function selection, finding that approximately one quarter of file-hit records yield a changed-function hit under the tested procedures. Separate repair experiments examine associations between function correctness and resolution under file-given and function-conditioned inputs. The paper advocates evaluating file recall alongside path sensitivity, function localization, and downstream repair outcomes.

Critical strengths

The paper is well motivated: distinguishing successful file retrieval from useful code localization addresses an important evaluation gap. Its clearest contribution is the controlled comparison of readable misleading paths and opaque identifiers while keeping the remaining inputs fixed. The main research questions are easy to follow, and the matched code-input controls, multi-seed factorial experiment, and historical-rename analysis provide complementary evidence. The authors also carefully distinguish associations from causal effects, disclose unsuccessful checks, and avoid interpreting statistically unresolved comparisons as equivalence.

Critical weaknesses

My main concern is whether the function-transfer analysis sufficiently separates limitations of file localization from limitations of the downstream scoring procedure. The shared scorers receive a file summary and candidate function name, without an explicit candidate-function-body input (Section 5.2 and Appendix C.1). Replacing the function name with an opaque identifier therefore removes a key means of distinguishing functions within the same file without supplying corresponding body-level evidence. Moreover, transfer for the same Codeaware file-hit set falls from 28.5% with its own scorer to 8.3% with the fixed-neutral scorer. Although the authors acknowledge this dependence, it limits the broader significance of the low transfer rates and unresolved comparisons. A stronger shared scorer with access to candidate function bodies, together with function-candidate coverage, would make this analysis more convincing. Second, the practical significance of the path interventions remains only partially established. The primary experiments concern pointwise rerankers with short code previews and deliberately broken path–content correspondence. Historical renames produce much smaller top-1 losses, while higher-cutoff recall improves. The supplementary frontier-model checks and unsuccessful native SweRank reproduction do not establish whether the central findings hold for competent systems that actively inspect repository code. A matched evaluation with stronger code access or an established agentic localizer would help distinguish a broadly consequential failure mode from sensitivity specific to the tested interface. Finally, the connection to repair remains incomplete. The strongest positive associations arise when the fixer must rewrite the supplied function, making failure on an incorrect function partly inherent to the interface. The less constrained file-given comparison remains inconclusive, and the released-system checks are mixed. These limitations are transparently acknowledged, but they leave open whether the proposed diagnostics provide useful predictive information beyond file accuracy in realistic repair workflows. A matched study on the same issues, allowing the fixer to inspect code and revise its edit location, would strengthen this part of the contribution.

Questions for authors

Could you clarify what function-specific code evidence the shared scorers receive, particularly after function names are masked? Would the conclusions hold with a stronger, function-body-aware scorer? Please refer to the first weakness.

Beyond the supplementary checks, can you establish whether the main path-sensitivity findings hold for a competent agentic localizer that actively inspects code?

Comments on the paper

1 comment

Thank you for the detailed rebuttal and additional experiments. While I appreciate the authors' efforts, I remain unconvinced that the new results sufficiently address my main concerns.

  1. The additional experiments demonstrate that even stronger models can be affected by misleading filenames. However, I remain concerned that the paper places disproportionate emphasis on sensitivity to artificial perturbations rather than demonstrating how these diagnostics improve real-world bug localization.

  2. The correction of the function-level hit rate from 24.8% to 45.0% is substantial. Although I appreciate the authors' transparency, such a large correction to a central experimental result raises concerns about the reliability of the original analysis.

The rebuttal provides additional evidence for the existence of path sensitivity, but my primary concerns regarding the practical significance, downstream implications, and strength of the conclusions remain. Therefore, I am inclined to maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.