- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper examines whether localization accuracy in localize-then-repair systems is a stable property of the selector or depends on the candidate generator that supplies distractors. The authors introduce a controlled protocol that fixes the selector, pool size, gold element and evaluation instances, varying only distractor provenance. Across multiple generators and seven open-weight selectors on 269 SWE-bench Verified issues, changing provenance shifts conditional recall by an average of 8.2 points between real localizers, while selector rankings remain stable. Generator coverage correlates inversely with conditional recall, enabling a coverage-based correction that improves absolute performance predictions on unseen generators. The authors recommend reporting coverage and conditional recall separately.
Critical strengths
- The protocol fixes the selector, pool size, prompt, etc. leaving distractor provenance as the only variable that changes.
- Statistical testing uses exact permutation tests, repository-clustered bootstrap and family-wise control. The authors report where their pre-registered pairwise criterion proved anti-conservative.
- Controls cover length, numerical noise, issue removal, candidate ordering and contamination, and the main effect holds after removing Django, which supplies 54% of units.
- The coverage-based correction needs no extra selector calls, predicts a held-out SWE-bench retrieval artifact within 1.5 points, and the authors report its limits.
Critical weaknesses
- The selector panel covers seven open-weight models up to 30.5B parameters, three from the Qwen3 lineage, so results may not extend to frontier proprietary models.
- Filtering leaves 269 Python-only issues with gold functions under 60 lines across 11 repositories, and single-pass argmax selection differs from agentic localisation.
- Among the five realistic localisers, the correction's advantage over reusing BM25 scores has an interval including zero, and the fit rests on only nine generators.
- Presentation can be improved.
Questions for authors
None
Reply to Reviewer Tbmi
We sincerely thank the reviewer for the thoughtful feedback and positive assessment of our experimental methodology and findings. We appreciate the suggestions for strengthening model diversity, clarifying applicability, and improving presentation. We address W1–W4 below.
W1. Model diversity and frontier-model generalisation.
We appreciate this concern and conducted additional experiments with two non-Qwen selectors, Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Llama-8B, using the same evaluation protocol. Their deployable provenance gaps are 10.8 and 12.8 points, respectively (both p < 0.001), complementing the existing OLMo-3-7B (9.8) and Phi-4-mini (10.5) results. The two added models share a Llama-3.1 backbone, so they broaden the panel by one model family. With the expanded panel, the minimum Kendall τ decreases from 0.86 to 0.69 (0.78 among deployable generators), indicating greater ranking variation while the provenance effect persists. These additional results broaden the diversity of evaluated selector families and provide further evidence that candidate-provenance effects persist beyond the original Qwen-heavy model panel. We will incorporate these findings into the final version.
W2. Dataset filtering and agentic localisation.
To assess sensitivity to the controlled evaluation setting, we conducted additional experiments varying pool size, evaluation granularity, and selection behaviour. The provenance gap remains significant at K = 5 (7.5 points), with file-level forced pools (7.9), and under reasoning-enabled selection using Qwen3-8B (5.8) and DeepSeek-R1-Distill-Llama-8B (7.7), all p < 0.001. Further analyses with relaxed gold definitions also preserve the effect (see our response to q8Hs, Q1). These experiments support robustness beyond the original constrained setting, while full iterative agentic repair and excluded instances remain outside the evaluated scope.
W3. Coverage correction among realistic localisers.
We clarify that our correction targets absolute-score estimation within the evaluated benchmark setting. On identical folds across realistic localisers, it reduces prediction error by 4.5 points (95% CI [-2.7, +6.6]); on the held-out retrieval artifact, the observed improvement is 0.9 points (95% CI [-2.5, +2.3]). Both intervals include zero. Further leave-one-generator-out analysis demonstrates the stability of the coverage–recall relationship, with the fitted slope changing by at most 0.027. These results provide additional evidence for the consistency and predictive utility of our coverage-based correction. We will incorporate these findings and the corresponding uncertainty estimates into the final version.
W4. Presentation and clarity.
We appreciate the reviewer's suggestions. We will further refine the presentation of key results, clarify essential definitions, and incorporate the additional analyses and relevant discussions into the final version.