acceptodds
Desk rejected at ICLR 2027

The Hidden Role of Candidate-Pool Provenance in Selector Evaluation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Localise-then-repair systems first retrieve a small set of candidate code locations and then select the most relevant one. Existing evaluations often interpret the resulting localisation accuracy as a property of the selector, implicitly assuming that this accuracy is stable across candidate generators. We show that this assumption does not generally hold. We introduce a controlled evaluation protocol that fixes the selector, candidate-set size, gold element, and evaluation instances, while varying only the generator that supplies the distractors. Across multiple candidate generators and 7 open-weight selectors on 269 SWE-bench Verified issues, changing candidate provenance substantially shifts conditional selection accuracy, with an average gap of 8.2 points between real localisers. Despite these shifts in absolute performance, selector rankings remain largely stable. We further find that generator coverage is strongly associated with selector difficulty, enabling a simple coverage-based adjustment that improves transfer of absolute performance estimates to unseen generators. These results show that selector accuracy should be reported together with the candidate generator on which it is measured, and motivate separating retrieval coverage from conditional selection performance in localisation evaluation.

Primary area:
reinforcement learning
Keywords:
SELECTOR EVALUATION

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TbmiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper examines whether localization accuracy in localize-then-repair systems is a stable property of the selector or depends on the candidate generator that supplies distractors. The authors introduce a controlled protocol that fixes the selector, pool size, gold element and evaluation instances, varying only distractor provenance. Across multiple generators and seven open-weight selectors on 269 SWE-bench Verified issues, changing provenance shifts conditional recall by an average of 8.2 points between real localizers, while selector rankings remain stable. Generator coverage correlates inversely with conditional recall, enabling a coverage-based correction that improves absolute performance predictions on unseen generators. The authors recommend reporting coverage and conditional recall separately.

Critical strengths
  1. The protocol fixes the selector, pool size, prompt, etc. leaving distractor provenance as the only variable that changes.
  2. Statistical testing uses exact permutation tests, repository-clustered bootstrap and family-wise control. The authors report where their pre-registered pairwise criterion proved anti-conservative.
  3. Controls cover length, numerical noise, issue removal, candidate ordering and contamination, and the main effect holds after removing Django, which supplies 54% of units.
  4. The coverage-based correction needs no extra selector calls, predicts a held-out SWE-bench retrieval artifact within 1.5 points, and the authors report its limits.
Critical weaknesses
  1. The selector panel covers seven open-weight models up to 30.5B parameters, three from the Qwen3 lineage, so results may not extend to frontier proprietary models.
  2. Filtering leaves 269 Python-only issues with gold functions under 60 lines across 11 repositories, and single-pass argmax selection differs from agentic localisation.
  3. Among the five realistic localisers, the correction's advantage over reusing BM25 scores has an interval including zero, and the fit rests on only nine generators.
  4. Presentation can be improved.
Questions for authors

None

1 comment
AuthorsOpenReview ↗

Reply to Reviewer Tbmi

We sincerely thank the reviewer for the thoughtful feedback and positive assessment of our experimental methodology and findings. We appreciate the suggestions for strengthening model diversity, clarifying applicability, and improving presentation. We address W1–W4 below.

W1. Model diversity and frontier-model generalisation.

We appreciate this concern and conducted additional experiments with two non-Qwen selectors, Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Llama-8B, using the same evaluation protocol. Their deployable provenance gaps are 10.8 and 12.8 points, respectively (both p < 0.001), complementing the existing OLMo-3-7B (9.8) and Phi-4-mini (10.5) results. The two added models share a Llama-3.1 backbone, so they broaden the panel by one model family. With the expanded panel, the minimum Kendall τ decreases from 0.86 to 0.69 (0.78 among deployable generators), indicating greater ranking variation while the provenance effect persists. These additional results broaden the diversity of evaluated selector families and provide further evidence that candidate-provenance effects persist beyond the original Qwen-heavy model panel. We will incorporate these findings into the final version.

W2. Dataset filtering and agentic localisation.

To assess sensitivity to the controlled evaluation setting, we conducted additional experiments varying pool size, evaluation granularity, and selection behaviour. The provenance gap remains significant at K = 5 (7.5 points), with file-level forced pools (7.9), and under reasoning-enabled selection using Qwen3-8B (5.8) and DeepSeek-R1-Distill-Llama-8B (7.7), all p < 0.001. Further analyses with relaxed gold definitions also preserve the effect (see our response to q8Hs, Q1). These experiments support robustness beyond the original constrained setting, while full iterative agentic repair and excluded instances remain outside the evaluated scope.

W3. Coverage correction among realistic localisers.

We clarify that our correction targets absolute-score estimation within the evaluated benchmark setting. On identical folds across realistic localisers, it reduces prediction error by 4.5 points (95% CI [-2.7, +6.6]); on the held-out retrieval artifact, the observed improvement is 0.9 points (95% CI [-2.5, +2.3]). Both intervals include zero. Further leave-one-generator-out analysis demonstrates the stability of the coverage–recall relationship, with the fitted slope changing by at most 0.027. These results provide additional evidence for the consistency and predictive utility of our coverage-based correction. We will incorporate these findings and the corresponding uncertainty estimates into the final version.

W4. Presentation and clarity.

We appreciate the reviewer's suggestions. We will further refine the presentation of key results, clarify essential definitions, and incorporate the additional analyses and relevant discussions into the final version.

Reviewer_q8HsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether the measured performance of a code-localisation selector remains stable when the upstream candidate generator changes. In two-stage localisation systems, an upstream generator determines which code elements enter a candidate pool, while a downstream selector chooses a location to edit. The authors argue that ordinary end-to-end accuracy entangles two distinct effects: whether the generator retrieves a gold location and how difficult the resulting distractors make selection. They introduce a controlled gold-forced pool protocol that fixes the issue, selector, pool size, gold code elements, prompting, and decoding while varying only the provenance of non-gold candidates. This isolates conditional recall as a function of the candidate generator. The evaluation covers 269 selected SWE-bench Verified issues, seven open-weight selectors, and a panel of lexical, dense, structural, LLM-based, issue-agnostic, and mixed candidate generators. Among five issue-aware localisers, transferring conditional recall between generators produces an average absolute gap of 8.2 percentage points, while the relative ranking of selectors remains largely stable. The authors further observe a strong negative association between generator coverage and conditional recall, and use coverage to adjust predictions for unseen candidate generators. This correction substantially improves prediction over the full generator panel, although its advantage is statistically unresolved when restricted to the five realistic localisers. A separate held-out SWE-bench retrieval artifact provides an additional, though relatively narrow, validation. Strengths. The work treats evaluation validity, rather than model accuracy improvement, as the research problem. Its gold-forced intervention isolates candidate provenance more carefully than comparisons of natural retrieval outputs, and the extensive controls support the existence of a meaningful generator-dependent shift. The distinction between unstable absolute scores and relatively stable selector rankings is useful, and the authors report several negative and boundary results rather than claiming universal transferability. Weaknesses. The findings are established primarily for a restricted function-level, fixed-size multiple-choice evaluation and are not yet shown to extend to iterative localisation or alternative notions of correct repair location. The practical advantage of coverage-based correction is uncertain within the realistic generator regime, and the external test remains close to the original benchmark and processing pipeline. A few reporting inconsistencies, including the sign convention for the transport gap and handling of pre-declared repeatability criteria, also require clarification.

Critical strengths
  1. The controlled intervention provides an informative separation of retrieval coverage and conditional selection difficulty. Sections 3.1–3.4 define a precise distinction between generator coverage, natural end-to-end accuracy, and conditional recall on gold-forced pools. Rather than comparing selectors on different naturally covered issue subsets, the protocol retains the same issues and byte-identical gold elements and changes only the non-gold candidates. This directly addresses the selection bias that would arise from restricting evaluation to issues each generator happens to cover. Appendix F substantiates the construction: unit-level coverage combined with forced conditional recall reproduces natural end-to-end accuracy with only 0.8 points of mean absolute residual, whereas conditioning on naturally covered units differs from the forced estimate by 8.0 points on average. This validation makes the proposed measurement more than an artificial pool perturbation.
  2. The primary provenance effect is backed by extensive controlled and sensitivity experiments. Across the seven selectors, Appendix J, Table 3 reports average conditional recall ranging from 62.1% on Mix-7 to 92.2% on Random pools despite the gold element being fixed. Crucially, the authors do not rely on this broad 30.1-point range, which includes deliberately non-deployable generators. Among five issue-aware localisers, the mean absolute cross-generator gap is 8.2 points, with an omnibus permutation result of p<0.001p<0.001 (Section 5.3). Appendix K shows that the effect persists after removing Dense, the dominant Django repository, or both; the last setting retains a 4.1-point gap (p=0.0076p=0.0076). The length, repeated-scoring, issue-removal, candidate-order, and within-repository controls in Appendices H and L further reduce plausible alternative explanations.
  3. The paper distinguishes what changes under generator shift from what remains stable, and reports the limitations of its correction. Section 5.5 and Appendix J, Figure 8 show that absolute conditional recall varies markedly, but selector rankings are comparatively robust: the lowest reported pairwise Kendall's τ\tau is 0.81 and the best selector is unchanged across the tested generators. This negative result is important because it prevents the evaluation critique from being overstated. Section 6 also tests a simple coverage-based adjustment: full-panel leave-one-generator-out prediction error falls from 10.2 to 2.2 points relative to direct BM25 score transfer, and the held-out retrieval artifact has 1.5 points of prediction error. The authors appropriately acknowledge that the advantage within the five realistic localisers has an interval including zero. These positive and negative results make the empirical contribution more complete.
Critical weaknesses
  1. The controlled selection task may not adequately represent the correctness and decision process of real software localisation. The study retains 269 of the 500 SWE-bench Verified instances and requires Python patches that map to at most four extractable functions or methods, each no longer than 60 lines (Section 4 and Appendix A). Every forced pool contains ten candidates, and the selector makes a single label choice using constrained next-token logits. These restrictions support identification, but they exclude longer or cross-language edits and much of the iterative inspection performed by software-engineering agents. More fundamentally, a location not edited in the historical gold patch is treated as a distractor even when it could potentially support an alternative valid fix. Harder pools containing semantically or structurally related code might therefore reflect ambiguity in the historical-patch label as well as genuine selector difficulty. The result is credible for matching the benchmark gold location, but its implications for deciding where a real repair should be made remain uncertain. A sensitivity check over candidate-pool size, alternative repair locations, or iterative selection would materially strengthen the practical interpretation.
  2. The coverage-only correction is not yet established as a generally useful way to transfer absolute accuracy between realistic generators. The high full-panel correlation between coverage and conditional recall (r=−0.987r=-0.987, R2=0.97R^2=0.97) partly spans deliberately extreme Random and History controls and constructed mixtures. While Appendix I shows that the fitted slope remains similar after excluding those generators, Section 6.3 reports that, among the five realistic localisers, the correction's estimated advantage over direct BM25 score transfer has an interval of [−1.7,+5.9][-1.7,+5.9] points, which includes zero. BM25 and Dense also have identical coverage (48.7%) but differ by about 4.5 points in conditional recall, demonstrating that coverage does not capture all relevant candidate difficulty. The single external test uses a SWE-bench BM25-derived retrieval artifact but retains the authors' element stage, gold construction, and selector interface (Appendix G). Finally, measuring coverage itself presumes gold-labelled evaluation issues; the proposed correction is not automatically available for unlabelled deployment requests. Thus, the calibration is a promising benchmark diagnostic rather than an established general transfer method.
  3. Some methodological claims and reporting conventions are not fully internally consistent. Equation (5) defines Δ(f;s→t)=R(f,s)−R(f,t)\Delta(f;s\to t)=R(f,s)-R(f,t), whereas Section 5.3 and Appendix J, Table 2 interpret the transport matrix as target performance minus source performance. For example, BM25 has approximately 68.0% conditional recall and Dense 63.6%, so Equation (5) implies a positive BM25-to-Dense value, while Table 2 presents a negative one. This leaves the meaning of signed transfer errors ambiguous even though absolute gaps are unaffected. Appendix E also reports that three of forty cells fail the pre-declared 98% repeatability criterion, with the worst at 97.4%, but still characterises the checks as having passed overall. The authors provide a plausible noise-floor argument for proceeding; however, that is a post hoc qualification to a stated criterion and should be presented as such. These issues are fixable, but precise definitions and faithful documentation are particularly important in a measurement-validity paper.
Questions for authors
  1. Regarding the real-world interpretation of conditional recall. How sensitive is the 8.2-point provenance gap to changing the candidate-pool size from K=10K=10 or allowing a selector to inspect code and revise its choice over multiple steps? Could the authors examine a sample of hard non-gold distractors to distinguish genuinely incorrect locations from plausible alternative repair sites? Would the main conclusion persist under a less restrictive gold definition or on the SWE-bench Verified instances excluded by the function-length and patch-mapping filters?
  2. Regarding the usefulness and transferability of the coverage correction. Can the authors test the coverage-based prediction on additional independently built candidate generators, different benchmarks, or generators with closely matched coverage but substantially different semantic hardness? What explains the uncertainty of its advantage among the five deployable localisers, and is there evidence that adding an inexpensive pool-difficulty feature improves over coverage alone? In a setting without known gold locations, how would the proposed correction obtain the coverage estimate required for prediction?
  3. Regarding metric definitions and pre-declared validation criteria. Should the transport gap be defined as R(f,t)−R(f,s)R(f,t)-R(f,s), as in the interpretation of Table 2, or should the table signs be reversed to match Equation (5)? Which signed convention is used in downstream statistical summaries? Also, was the 98% repeatability requirement in Appendix E intended to hold for each cell or on average? If it was a per-cell criterion, why were the three failing cells retained rather than reported explicitly as deviations from the pre-declared protocol, and do the main effects change when those cells are excluded?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer q8Hs (Part 1)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the opportunity to clarify our methodology, provide additional experimental evidence, and address the concerns raised below.


W1. Controlled evaluation and historical-patch gold.

We appreciate the reviewer's concern regarding real-world applicability and gold-label ambiguity. We clarify that our controlled protocol aims to isolate candidate-provenance effects rather than reproduce the full agentic repair process. To address these concerns, we conducted additional sensitivity analyses across pool sizes, reasoning-enabled selection, and relaxed gold definitions (Q1). The provenance gap persists across these settings, including after crediting plausible alternative fix sites (7.3 points, p < 0.001). We will clarify the scope of our findings and the remaining limitations concerning iterative agentic repair.


W2. Practical utility and transferability of coverage correction.

We appreciate the opportunity to clarify the intended use and limitations of our correction. It is designed for efficient absolute-score estimation in labelled evaluation settings, rather than gold-free deployment. The coverage–recall relationship remains strong across generator subsets (R² = 0.91–0.97). Among realistic localisers, correction reduces prediction error from 7.3 to 2.8 points on identical folds, although the confidence interval for this improvement includes zero. With a previously fitted calibration model, coverage estimates from 25/50 labelled issues yield prediction errors of 3.9/3.2 points without additional target-generator selector calls. We further examine matched-coverage generators, an inexpensive difficulty feature, and the applicability of our correction in Q2.


W3. Metric consistency and repeatability reporting.

We thank the reviewer for highlighting these reporting issues. We clarify that the sign reversal in Eq. (5) is a typographical error; Table 2 and our analyses consistently use target-minus-source differences, leaving all absolute-gap statistics unchanged. Regarding repeatability, we have further examined both correctness-level and chosen-label agreement, obtaining mean agreement rates of 99.1% and 98.4%, respectively. Sensitivity analyses further show that the main provenance effect remains significant after excluding the below-threshold deployable cells.

AuthorsOpenReview ↗

Reply to Reviewer q8Hs (Part 2)

Q1. Real-world interpretation of conditional recall.

We thank the reviewer for these constructive suggestions. To assess the robustness of our findings, we conducted additional sensitivity analyses across pool sizes, reasoning-enabled selection, and relaxed gold definitions, as summarised in Table R1.

(a) Pool size and reasoning-enabled selection. The provenance gap remains significant at K = 5 and under free-form reasoning with Qwen3-8B and DeepSeek-R1-Distill-Llama-8B (all p < 0.001; Table R1). The difference between K = 5 and K = 10 is not statistically resolved (95% CI [-1.69, +0.15]). Reasoning reduces the gap for both selectors, with a statistically significant reduction for DeepSeek (difference -6.0 points, 95% CI [-7.3, -1.2]). These experiments examine reasoning-enabled single-step selection rather than full iterative agentic repair.

(b) Gold ambiguity and relaxed definitions. We examined 96 high-consensus non-gold distractors through LLM-assisted annotation, identifying 25 plausible alternative fix sites. Crediting these alternatives preserves a significant 7.3-point provenance gap (p < 0.001). The effect also persists across three relaxed gold definitions and separately constructed file-level forced pools (Table R1). Notably, under gold-file credit, the raw gap decreases from 8.8 to 5.4 points, whereas the chance-corrected gap decreases only from 10.5 to 9.0 points, indicating that changes in the number of acceptable candidates contribute to the observed reduction. These analyses assess potential gold ambiguity without treating plausible alternatives as verified repairs.

(c) Excluded instances and scope. The 231 excluded SWE-bench Verified instances were not re-evaluated, as relaxing the extraction constraints requires reconstructing the element universe. We therefore restrict our empirical claims to the evaluated Python function-level setting and will clarify this scope in Limitations.

Together, these analyses show that the provenance effect persists across the tested variations in pool size, selection procedure, and gold definition, rather than depending solely on the original constrained protocol.

Table R1. Robustness of candidate-provenance effects across alternative evaluation settings.

AnalysisEvaluation settingProvenance gap (pts)Reference gap (pts)Gap significance
Pool sizeK = 57.58.9 (K = 10)p < 0.001
ReasoningQwen3-8B, free-form5.87.3 (constrained)p < 0.001
ReasoningDeepSeek-R1-8B, free-form7.713.7 (constrained)p < 0.001
Gold ambiguityCredit 25 plausible alternatives7.38.8 (strict)p < 0.001
Gold definitionAny candidate in a gold file5.48.8 (strict)p < 0.001
Gold definitionAny method in a gold class7.18.8 (strict)p < 0.001
Gold definitionName-based call-graph neighbour7.08.8 (strict)p < 0.001
Evaluation granularityFile-level forced pools7.98.9 (function-level)p < 0.001

Note: Reported p-values test whether each provenance gap differs from zero, not whether it differs from the reference gap. Reasoning-enabled results are selector-specific; other rows aggregate across selectors. The 8.9-point reference comes from matched re-evaluation, whereas 8.8 points is the corrected strict-gold aggregate.

AuthorsOpenReview ↗

Reply to Reviewer q8Hs (Part 3)

Q2. Usefulness and transferability of the coverage correction.

We thank the reviewer for these suggestions. We clarify the intended scope of our correction and summarise the relevant evidence and additional analyses in Table R2.

(a) Transferability and matched coverage. Our evaluation focuses on SWE-bench Verified under a shared element-extraction and selector protocol. The held-out SWE-bench_bm25_13K artifact provides an additional retrieval source but retains our element stage. Notably, BM25 and Dense have identical coverage (48.7%) yet differ in conditional recall (3.7 points, p = 0.015), consistently across all seven selectors. This confirms that coverage alone does not fully capture candidate-pool difficulty.

(b) Uncertainty and difficulty features. On identical folds across deployable localisers, our correction reduces prediction error from 7.3 to 2.8 points (gain 4.5, 95% CI [-2.7, +6.6]). This paired analysis corrects the unequal fold sets used in the original comparison. Limited training generators and repository imbalance may contribute to the uncertainty. We also examined a simple gold-file-derived difficulty feature: the number of distractors from the gold file. Adding it reduces prediction error from 2.16 to 1.70 points, although the incremental improvement remains statistically unresolved.

(c) Coverage estimation and evaluation cost. We clarify that coverage estimation requires labelled issues. With a previously fitted calibration model, however, our correction avoids additional selector calls for the target generator. Using 25/50/100 labelled issues, prediction errors are 3.9/3.2/2.7 points, versus 6.4/4.5/2.9 from direct selector evaluation at the same labelled-issue budgets. These within-dataset results suggest a practical advantage for low-budget performance estimation. We will clarify the labelled-data requirement and generalisation scope in the revision.

Table R2. Additional analyses of coverage correction.

AnalysisComparisonResult (pts)95% CI of difference
Matched coverageBM25 vs. Dense (48.7% each)Recall gap: 3.7[+1.1, +8.8]
Realistic localisersBM25 transfer vs. correction7.3 → 2.8[-2.7, +6.6]
Difficulty featureCoverage vs. Coverage + difficulty2.16 → 1.70[-1.5, +1.4]
Budget: 25 labelsDirect evaluation vs. correction6.4 → 3.9—
Budget: 50 labelsDirect evaluation vs. correction4.5 → 3.2—
Budget: 100 labelsDirect evaluation vs. correction2.9 → 2.7—

The first row reports conditional-recall differences; all remaining rows report prediction errors (lower is better). Confidence intervals correspond to the direction of the reported difference or improvement. Budget experiments use within-dataset subsampling.


Q3. Metric definitions and the repeatability criterion.

We thank the reviewer for identifying these reporting inconsistencies and would like to clarify both points.

(a) Sign convention. The sign reversal in Eq. (5) is a typographical error. The intended definition is Δ(f;s→t)=R(f,t)−R(f,s)\Delta(f;s\rightarrow t)=R(f,t)-R(f,s) (target minus source), consistent with Table 2, Fig. 5b, and Sec. 5.3. All downstream transport-gap statistics use absolute differences, so the headline provenance gap, permutation tests, family-wise thresholds, and prediction errors remain unchanged.

(b) Repeatability. We clarify that the pre-declared 98% criterion did not explicitly distinguish between per-cell and aggregate agreement. We therefore further examined both correctness-level and chosen-label agreement across 40 repeated evaluation cells (4 selectors × 10 generators):

  • Correctness agreement: 99.1% on average; 3/40 cells below 98% (minimum 97.4%).
  • Chosen-label agreement: 98.4% on average; 14/40 cells below 98% (minimum 96.3%).

The latter more directly reflects the reproducibility of selector decisions. We acknowledge that the original "all passed" description was overly broad and that the noise-floor interpretation was post hoc.

To examine the impact of these deviations, we conducted additional sensitivity analyses. The deployable provenance gap is 8.77 points using all cells, 8.84 after excluding the three correctness-level failures (p < 0.001), and 9.52 after excluding the seven below-threshold deployable cells at the chosen-label level. These results indicate that the main provenance effect remains robust under both exclusions.

We will revise the relevant descriptions accordingly in the final version.

Reviewer_B9Y5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the provenance of a candidate pool affects the measured accuracy of a frozen code-localization selector. It holds the evaluation issues, gold code elements, pool size and selector protocol fixed while changing the source of distractors. On 269 filtered SWE-bench Verified issues and seven open-weight selectors, it reports an average 8.2-point conditional-recall gap among five issue-aware localizers, while selector rankings remain relatively stable. It also proposes a simple coverage-based correction for transferring absolute scores, evaluated by leaving out generators and by using an external BM25 file-retrieval artifact. The contribution consists mainly of a controlled evaluation protocol, an empirical characterization of generator dependence, and a score-transfer heuristic.

Critical strengths
  1. A useful controlled quantification of candidate-pool effects. Fixing the issue set, gold content, candidate count, selector and scoring procedure makes the distractor-source effect interpretable. The comparison among five issue-aware localizers, together with the length and repository-removal controls, provides useful evidence that absolute selector scores should be reported with candidate provenance (Sections 3-5, Appendices H and K).

  2. The analysis distinguishes absolute-score shifts from ranking stability and acknowledges limits. The paper reports that selector rankings are comparatively stable and that the correction advantage among realistic localizers has an interval containing zero. The natural-pool checks and held-out retrieval artifact are constructive attempts to examine applicability. This transparency helps delimit the empirical finding and avoids presenting every positive point estimate as an established practical improvement (Sections 5.5-6, Appendices F-I).

Critical weaknesses
  1. The incremental conceptual and methodological contribution is limited. The main finding, that more confusable alternatives change selection accuracy, is closely related to the retriever-dependent negative hardness discussed in Section 2.2. The controlled localization study usefully quantifies this effect, but adds limited mechanism-level understanding or broadly validated predictive structure beyond the tested setting. The principal methodological additions are the forced-pool protocol and a simple coverage correction. A new localization or training algorithm is not a prerequisite for acceptance; the issue is whether the measurement establishes enough distinct knowledge beyond the existing candidate-difficulty perspective.

  2. The demonstrated practical consequences and utility of the proposed correction remain limited. Although absolute scores shift substantially, selector rankings remain largely stable and choosing the best selector on one generator incurs zero selection regret across the reported ordered generator pairs (Section 5.5). The study therefore does not establish that provenance changes the model-selection decision in this panel. For the five realistic localizers, the coverage correction reduces point-estimate prediction error from 6.5 to 2.5 points, but its reported advantage of 3.9 points has an interval of [-1.7, +5.9], including zero. The external test yields errors of 1.5 versus 2.5 points while retaining the study's own element stage. These results support a reporting caution, but provide limited evidence that the proposed correction delivers a reliable, consequential improvement in realistic deployment evaluation. The distinction between forced-pool recall and natural covered-issue accuracy further limits a direct deployment interpretation (Section 6; Appendices F-G).

Questions for authors

What specific new conclusion about selector behavior does the controlled experiment establish beyond the retriever-dependent negative-hardness effects discussed in Section 2.2? Please distinguish the value of quantification in code localization from a new explanatory mechanism.

Given the stable rankings and zero model-selection regret in the reported panel, which substantive evaluation or deployment conclusions are changed by the proposed protocol or correction? An example or reanalysis within the existing study would help establish the practical significance.

Please provide the uncertainty for the external generator's 1.0-point prediction-error improvement and clarify what evidence supports usefulness of the correction among realistic localizers, where the reported advantage interval contains zero. What scope of applicability is justified without a broader retrieval range?

Please state separately the forced-pool recall and natural covered-issue accuracy, and explain the conditions under which the correction predicts a quantity relevant to the natural pipeline. Please relate this explanation to the aggregate-product residuals reported in Appendix F.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer B9Y5 (Part 1)

We thank the reviewer for the thoughtful and constructive feedback, which has helped us further clarify the contributions and significance of our work.


W1. Limited contribution beyond retriever-dependent negative hardness.

While negative hardness is well known, our controlled study establishes three distinct findings about evaluation validity:

  1. Identification. Conventional covered-issue accuracy differs from controlled conditional recall by 7.0 points on average, with weak agreement in cross-generator gaps (r ≈ 0.4). By fixing the selector and gold element, our protocol isolates candidate-provenance effects that conventional evaluations confound (App. F).

  2. Structure. Generator effects are nearly additive across selectors: interactions explain only 2.5% of the variance across 63 selector–generator cells. This explains why absolute scores shift while rankings remain stable (minimum τ = 0.86). The same generator effects predict two additional selectors with RMS errors of 2.0 and 3.6 points.

  3. Equal coverage, unequal difficulty. BM25 and Dense achieve identical coverage (48.7%) but differ in conditional recall by 3.7 points [+1.1, +8.8] (p = 0.015). Thus, retrieval coverage alone cannot determine downstream selection difficulty.

These findings go beyond the known hard-negative effect by establishing how candidate provenance changes the validity and transferability of selector evaluations.


W2. Practical consequences and the utility of the correction.

We clarify that zero model-selection regret does not imply limited evaluation significance. Although selector rankings remain stable, substantial absolute-score shifts demonstrate that performance estimates are not directly transferable across candidate generators. Our protocol addresses this distinct evaluation problem by separating retrieval coverage from conditional selection performance and identifying confounding in conventional comparisons (Q2). Importantly, uncertainty about the correction's effectiveness does not undermine the observed provenance dependence: coverage-based calibration improves prediction over the full generator range, although its advantage among realistic localisers remains statistically unresolved (Q3). We will also clarify the scope of the held-out file-ranking test and the distinction between forced-pool recall and natural-pipeline accuracy (Q4). Thus, our contribution extends beyond reporting conventions to clarifying when absolute selector scores can be meaningfully compared and transferred across candidate generators.


Q1. What new conclusion, beyond Sec. 2.2? Quantification versus mechanism.

The key distinction is between studying how hard negatives affect predictions and establishing how candidate provenance affects evaluation validity. Prior work identifies the former; our controlled protocol establishes three additional findings (W1): conventional covered-issue comparisons confound provenance with issue selection; generator effects are nearly additive across selectors, explaining stable rankings despite shifting absolute scores; and identical retrieval coverage can conceal different selection difficulties. These findings establish new properties of evaluation, not merely another measurement of negative hardness. Importantly, the provenance effect is substantial (8.8-point average gap among deployable localisers, permutation p < 0.001) and remains significant after removing Dense or django (App. K). Thus, our contribution is not a new hard-negative mechanism, but a controlled account of when selector scores are comparable and transportable across generators.

AuthorsOpenReview ↗

Reply to Reviewer B9Y5 (Part 2)

Q2. Which conclusions change, given stable rankings and zero regret?

We clarify that relative model selection and absolute performance evaluation are distinct objectives. Zero model-selection regret shows that the best selector remains unchanged across generators in our panel; it does not imply that absolute scores are transferable. Indeed, the coexistence of stable rankings and substantial absolute-score shifts is a central finding, with four concrete evaluation consequences:

  1. Absolute-score validity. The same Qwen3-VL-30B-A3B achieves 78% conditional recall on Dense pools versus 91% on Struct pools, with an average 8.8-point transport gap among deployable localisers. Absolute scores are therefore generator-dependent.
  2. Evaluation confounding. Natural covered-issue accuracy differs from controlled conditional recall by 7.0 points on average, and their pairwise generator gaps correlate at only (r\approx0.4). Conventional comparisons can misattribute generator effects.
  3. Retrieval evaluation. BM25 and Dense have identical coverage (48.7%) but differ in conditional recall by 3.7 points [+1.1, +8.8]. Recall@k alone therefore misses downstream selection difficulty.
  4. Score transport. Coverage-based calibration reduces cross-generator prediction error from 10.9 to 2.1 points over the full generator range. Thus, stable rankings preserve the model-selection decision, but not the validity of absolute performance claims or cross-generator comparisons. Our protocol identifies precisely which evaluation conclusions remain valid under candidate-provenance shifts and which require provenance-aware measurement.

Q3. Uncertainty of the external gain; evidence among realistic localisers; scope.

(a) External generator. To quantify uncertainty, we perform 1,000 paired bootstrap resamples, refitting each selector's coverage calibration model in every resample. On the held-out SWE-bench_bm25_13K artifact, our correction reduces prediction error from 2.6 to 1.2 points, an improvement of 1.4 points (95% CI [-0.3, +2.5]). The tighter interval provides a more precise estimate of the gain, with a negative lower bound of only 0.3 points, although it still includes zero. We will report these results in App. G.

(b) Realistic localisers. On identical folds across four non-BM25 deployable localisers, our correction reduces prediction error from 7.3 to 2.8 points (gain 4.5, 95% CI [-2.7, +6.6]), correcting the paper's comparison across unequal fold sets (35 vs. 28). While the interval includes zero, the point estimate favours our method. Under limited evaluation budgets, a previously fitted calibration model achieves errors of 3.9/3.2 points using coverage from just 25/50 labelled issues, versus 6.4/4.5 from direct selector evaluation, without additional target-generator selector calls. At 100 issues, the errors converge (2.7 vs. 2.9). These within-dataset results highlight the correction's potential for cost-efficient performance estimation.

(c) Scope. Our experiments establish the applicability of coverage-based calibration within the evaluated setting: SWE-bench Verified, function-level candidates, K = 10, and constrained-label selectors. Importantly, the coverage–recall relationship remains strong across generator subsets (R² = 0.91–0.97, App. I), including fits restricted to deployable localisers. Their coverage spans 17.5–48.7%, encompassing the held-out artifact (44.2%). These findings support prediction within the observed coverage regime.


Q4. Forced-pool recall versus natural covered-issue accuracy, and App F.

Here they are separately for the deployable generators, pooled over the 7 selectors (%):

GeneratorCoverageForced R, all 269Natural acc., covered issues (n)End-to-end
BM2548.767.572.4 (131)35.3
Dense48.763.769.0 (131)33.6
Struct17.581.273.6 (47)12.9
LLM-A30.576.184.0 (82)25.6
LLM-B46.169.978.9 (124)36.4

Our correction predicts forced-pool conditional recall RR across all issues, separating selector performance from retrieval coverage. Natural covered-issue accuracy is distinct but closely matches forced RR evaluated on the same covered issues (BM25: 72.4% vs. 74.7%). At the issue level, the product covuRu\mathrm{cov}_u R_u approximates end-to-end accuracy with a mean absolute residual of 0.7 points (App. F). In contrast, the aggregate product cov×R\mathrm{cov}\times R underestimates end-to-end accuracy by 2.0 points on average (mean absolute residual: 2.2), reflecting coverage–difficulty covariance and differences between forced and natural selection outcomes. Thus, our correction provides a provenance-aware estimate of conditional selection performance, while recovering natural-pipeline accuracy requires accounting for these additional factors. We will clarify this distinction in Sec. 6 and include the corresponding comparisons in App. F.

AuthorsOpenReview ↗

Hugely appreciate the suggestions and looking forward to engaging further!

Thank you again for your constructive feedback. We have responded to your concerns point by point and hope our responses help clarify the issues you raised. If you have any further questions, we would be happy to provide additional clarification. Thank you again for your time and consideration!

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Request for Reassessment

Dear Reviewer,

We would appreciate your feedback on our rebuttal when you have an opportunity. Please let us know if any concerns remain or further clarification would be helpful.

If our responses address your concerns, we would be grateful if you could reconsider your assessment and score.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Evidence

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback. As the discussion period closes, we summarize the recognized strengths and additional rebuttal evidence.

Strengths recognized by the reviewers.

  • Problem formulation. Our work identifies candidate-pool provenance as a key factor in selector evaluation, separating retrieval coverage from conditional selection performance (B9Y5, q8Hs, Tbmi).
  • Controlled protocol. Fixing selectors, issues, gold elements, and pool size while varying distractor provenance isolates generator-dependent effects (B9Y5, q8Hs, Tbmi).
  • Experimental evidence. Across seven selectors and 269 SWE-bench Verified issues, we observe an 8.2-point conditional-recall gap among realistic localisers, supported by statistical and sensitivity analyses (B9Y5, q8Hs, Tbmi).
  • Evaluation insights. Absolute scores shift across generators while selector rankings remain relatively stable. Coverage-based calibration further examines score transferability and its limitations (B9Y5, q8Hs, Tbmi).

Reviewer B9Y5. Our rebuttal addresses the reviewer's concerns:

  • Beyond negative hardness (W1, Q1). Conventional covered-issue accuracy differs from controlled recall by 7.0 points. Generator–selector interactions explain only 2.5% of variance. BM25 and Dense have identical coverage (48.7%) but differ in recall by 3.7 points (p = 0.015), establishing distinct evaluation properties.
  • Practical significance (W2, Q2). Despite stable rankings, absolute scores vary substantially: Qwen3-VL-30B-A3B achieves 78% recall on Dense versus 91% on Struct. Calibration reduces full-panel prediction error from 10.9 to 2.1 points.
  • Correction uncertainty (Q3). External prediction error decreases from 2.6 to 1.2 points (95% CI of improvement: [-0.3, +2.5]); among realistic localisers, from 7.3 to 2.8 (95% CI [-2.7, +6.6]). Both intervals include zero.
  • Metric interpretation (Q4). We distinguished forced-pool recall from natural covered-issue accuracy. Issue-level coverage-weighted recall approximates end-to-end accuracy with a 0.7-point mean absolute residual, while aggregate estimates require accounting for coverage–difficulty covariance.

Reviewer q8Hs. We added robustness experiments and methodological clarifications:

  • Evaluation robustness (W1, Q1). Provenance effects remain significant with K = 5, reasoning-enabled selection, relaxed gold definitions, and file-level pools (all p < 0.001). Crediting plausible alternative fix sites preserves a 7.3-point gap.
  • Correction utility (W2, Q2). With 25/50 labelled issues, calibration yields prediction errors of 3.9/3.2 points versus 6.4/4.5 for direct evaluation. The coverage–recall relationship remains strong (R² = 0.91–0.97), although realistic-localiser gains remain statistically unresolved.
  • Consistency and repeatability (W3, Q3). We corrected the transport-gap sign convention and examined correctness and chosen-label agreement (99.1% and 98.4%). Excluding below-threshold cells preserves the significant provenance effect.

Reviewer Tbmi recognized the controlled design, statistical rigor, and robustness checks, assigning a rating of 3 (Weak Acceptance). We further addressed the concerns:

  • Model diversity (W1). Two additional non-Qwen selectors, Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Llama-8B, yield gaps of 10.8 and 12.8 points (both p < 0.001). The expanded panel shows greater ranking variation while confirming provenance dependence.
  • Evaluation scope (W2). Pool-size, reasoning-enabled, file-level, and relaxed-gold experiments preserve significant effects, while full iterative agentic repair remains outside our scope.
  • Correction and presentation (W3–W4). We clarified the correction's labelled-benchmark scope, reported uncertainty, and committed to improving definitions, presentation, and limitations.

Together, these analyses strengthen our finding that candidate-pool provenance substantially affects absolute selector scores despite relatively stable rankings. Expanded evaluations and robustness checks further clarify the applicability and limitations of provenance-aware evaluation.

We thank the reviewers and Area Chair for their time and consideration and hope our additional evidence will inform the final assessment.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.