- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper proposes a set of diagnostic evaluations for interpreting bug localization results. It argues that file-level recall can obscure two important limitations: a reranker may rely heavily on file paths even when code is available, and identifying the correct file does not necessarily identify the function that needs to change. Using fixed candidate pools, the authors test rerankers with readable aliases drawn from unrelated repositories, opaque path identifiers, path exchanges, and historical file names. On SWE-bench Lite, the PathAlias intervention reduces Recall@1 by 21.8 percentage points for a Qwen2.5-14B reranker trained without path augmentation, compared with 4.9 points for a path-augmented version. On GREPO, only 585 of 2,356 file-hit records (24.8%) also identify a changed function using each system’s own function scorer. Finally, separate repair experiments show that supplying the correct function is strongly associated with successful resolution when the fixer is instructed to rewrite that function, while the evidence from file-given repair remains inconclusive.
Critical strengths
-
It reveals distinct underlying behaviors associated with identical scores. Under the original path setting, the Recall@1 scores for the two 14B models—one with code and one without—were comparable (57.0% vs. 55.7%); however, when the paths were replaced with unreadable identifiers, the figures shifted to 39.7% and 0.3%, respectively. This compellingly demonstrates that standard file-level scores are insufficient to determine whether a model possesses the capability to utilize code.
-
The paper analyzes the complete "file-to-function" workflow, compares results using a shared function-level scorer, and examines two types of repair inputs: "file-provided" and "function-provided." Furthermore, the authors avoid interpreting confidence intervals that span zero as evidence of equivalence between methods, nor do they mistake correlations observed in the repair experiments for causal effects.
Critical weaknesses
-
There is a lack of experiments on state-of-the-art LLMs; I believe the issues raised by the authors do not pose significant obstacles for these advanced models.
-
I feel this evaluation method somewhat deviates from the core objective of the task. Placing too much emphasis on the effects of path intervention shifts the focus away from the benchmark's primary evaluation criteria.
-
The shared function scorer utilizes file paths, summaries, and function names but does not read the function bodies; furthermore, comparisons are restricted to instances where both file rankers correctly identify the target file. Consequently, this does not rule out the possibility that code-assisted file ranking could yield downstream benefits when integrated with more powerful function locators or a complete workflow.
Questions for authors
-
Please conduct experiments using additional state-of-the-art models.
-
Please explain the significance of this evaluation for the task at hand.
-
Please revise the wording of the paper as much as possible; I had to read it twice to understand the task involved, and I do not believe other readers will have the patience to do the same. Additionally, the figures in the paper look overly "AI-generated."
Thank you for the detailed rebuttal and additional experiments. While I appreciate the authors' efforts, I remain unconvinced that the new results sufficiently address my main concerns.
The additional experiments demonstrate that even stronger models can be affected by misleading filenames. However, I remain concerned that the paper places disproportionate emphasis on sensitivity to artificial perturbations rather than demonstrating how these diagnostics improve real-world bug localization.
The correction of the function-level hit rate from 24.8% to 45.0% is substantial. Although I appreciate the authors' transparency, such a large correction to a central experimental result raises concerns about the reliability of the original analysis.
The rebuttal provides additional evidence for the existence of path sensitivity, but my primary concerns regarding the practical significance, downstream implications, and strength of the conclusions remain. Therefore, I am inclined to maintain my original rating.