- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the redundancy of similarity-based corrective demonstrations in LLM code repair, which can limit the complementary repair guidance provided by retrieved examples. It proposes IT-CES, a training-free example selection method that combines the problem statement, buggy code, and execution failure signal into a diagnostic query to retrieve relevant historical repair cases. It then applies distributional calibration to construct a candidate support set and greedily selects complementary demonstrations using a log-determinant objective that balances relevance and diversity. On a fixed hard set of 170 problems from three data sources, IT-CES achieves first-attempt repair success rates of 28.8% and 28.2% with three and five demonstrations, respectively, outperforming the strongest retrieval baselines in the main comparison by 4.7 and 5.9 percentage points. Ablation studies, evaluations across different model configurations, and multi-round dynamic repair experiments further support its effectiveness within the tested settings.
Critical strengths
The paper’s core strength lies in conditioning corrective example selection on the specific failure state of the current code while jointly considering relevance and complementarity at the set level. By combining the problem statement, buggy code, and execution failure signal, IT-CES retrieves targeted historical cases and uses calibration and log-determinant selection to reduce redundancy, allowing a limited context budget to cover more repair cues. The method requires no additional model training, incurs low reranking overhead, and achieves consistent improvements across the three evaluated data sources, different model configurations, and multi-round repair settings, demonstrating practical value.
Critical weaknesses
The paper’s core weakness is that it does not sufficiently establish its substantive novelty over existing example selection methods. The algorithm primarily combines diagnostic retrieval, candidate filtering, relevance weighting, and an existing log-determinant selection framework. The information-theoretic and Gaussian-process analyses provide design motivation but do not establish a theoretical connection between the selection objective and code repair success. Consequently, the contribution relies primarily on empirical improvements, while the evaluations remain concentrated on the same set of 170 difficult problems, leaving the applicability and robustness of these improvements in broader settings insufficiently established.
Questions for authors
- Why does calibrated Top-K produce different results?According to the method description, standardization and softmax preserve the similarity ranking, while support filtering retains a prefix of that ranking containing at least candidates. Calibrated Top-K should therefore select the same examples as ordinary Top-K. Why does Table 3 report an increase from 32 to 34 successful repairs? Could you provide the selected example IDs, their ordering, and the final prompts for both configurations to explain this difference?
- What is the substantive contribution beyond existing log-determinant selection methods? When using identical diagnostic representations, candidate pools, and hyperparameter tuning budgets, what distinguishes IT-CES from KITE or a standard quality–diversity DPP selector? Could you more clearly isolate the contributions of diagnostic representations, candidate support calibration, and quality weighting?
- Does distributional calibration help primarily by reducing the candidate pool? Could you add a control that retains the top candidates by raw similarity, with the candidate count matched to IT-CES’s support size for each query? This would help determine whether the gains arise from adaptive probability-based filtering or simply from excluding less relevant candidates.
- What direct evidence connects geometric coverage to repair effectiveness? The information-theoretic and Gaussian-process analyses primarily provide design motivation. Could you examine whether log-determinant gains correlate with repair success after controlling for relevance and demonstration count, and analyze failure cases where geometric complementarity does not translate into useful repair guidance?
- Do the results generalize to independent test sets and model-generated errors? The current 170-problem cohort is selected based on Qwen3-Coder-Next’s zero-shot failures, while the repair targets are fixed historical buggy submissions. Could you evaluate on an independent test set without this model-specific filtering, as well as on erroneous code generated by each evaluated model, to assess how test-set selection and error provenance affect the conclusions?
- Does dynamic multi-round repair outperform equally dynamic strong baselines? The multi-round experiments primarily compare Dynamic Top-K with Dynamic IT-CES. Could you include MMR and KITE baselines that also retrieve and select new demonstrations after every failed round, while controlling model calls and context token budgets? This would help distinguish the contribution of dynamic updating from that of IT-CES’s selection strategy.
4 comments
Response to Reviewer Sjwe (2)
Question 2 — Contribution and controlled comparisons
IT-CES specifies a failure-conditioned quality–diversity DPP and tests it against existing selectors under shared inputs and tuning budgets. It selects from the observable failure tuple without the query's correct program or generator calls. Score-shape-dependent support, raw-cosine quality weights and a diagnostic Gram kernel define the rule. KITE additionally optimizes query-prediction error; MMR uses pairwise redundancy. These are distinct preferences: with selected vectors e1,e2 and equal-relevance candidates (e1+e2)/sqrt(2) and .8e1+.6e3, MMR prefers the first (maximum cosine .7071<.8), whereas our shifted Gram kernel gives conditional residual variances .0286<.1867 and log-det prefers the second. This zero-jitter example explains the subset preference; repair outcomes provide the empirical test below. Section 3.2 (pp. 4–5) and Appendix A.2 (p. 13) state the geometric criterion and give its marginal, including jitter.
The component interventions are explicit: diagnostic/code-only Top-K repairs 32/26 problems; DPP configurations (alpha,p)=(1,1),(1,.8),(2,1),(2,.8) repair 36/40/43/48. Alpha=1 retains quality weighting. These are descriptive results, not independently established component effects (Table 4, p. 9; Appendix D.2, pp. 20–21).
Calibrated MMR/KITE share IT-CES's diagnostic embeddings, top-200 retrieval, p=.8 support, generator, prompt and k=5. IT-CES/MMR/KITE each receive 18 trials on the same separate 60 validation problems, with all 54 results disclosed in Table 6 (p. 18). This matches trial counts, not parameter spaces. On 800 independent problems:
| Method | Repairs/800 | IT-only / comparator-only | Holm p |
|---|---|---|---|
| IT-CES | 457 | — | — |
| Calibrated MMR | 429 | 96 / 68 | .069361 |
| Calibrated KITE | 427 | 88 / 58 | .048297 |
| Fixed top-42 IT-CES | 438 | 87 / 68 | .147998 |
All three exact McNemar tests belong to one Holm family (Tables 2/13, pp. 8/22). KITE compares complete objectives; fixed42 changes support size alone. KITE already includes log-det. With diagnostic inputs and support held fixed, the gain supports the specified quality–diversity objective relative to KITE’s query-reconstruction objective, not the mere presence of diversity. MMR changes quality handling and subset scoring together, so it cannot identify log-det’s separate contribution. Fixed42 isolates support formation, with inconclusive evidence for an adaptive-support advantage. Thus the complete-objective comparison and component attribution answer different questions; the latter remains unresolved, rather than establishing zero component effects. “Calibrated” names the support operation, not an independently verified gain. Table 1 (p. 8) now displays original and stronger shared-support 170-problem results together; the controlled tests remain nonsignificant (Table 11, p. 22). All 800 paired outcomes are disclosed in Appendix F (p. 24).
Question 3 — Matched candidate count
Your proposed matched-count control is exactly equivalent: raw top-M with M equal to each query's nucleus size gives exactly the same support. Calibration changes cardinality, not ranking (Section 3.3, pp. 5–6). We therefore compare fixed top-42, retaining IT-CES's quality weights, kernel and greedy rule while changing only support formation.
Fixed42 repairs 438/800 versus IT-CES's 457/800: +2.375 percentage points, pointwise 95% CI [−.625,5.375], Holm p=.147998. This control isolates support formation while keeping the objective fixed. The interval includes zero and the corrected test is nonsignificant, establishing neither superiority nor equivalence. Adaptive support therefore remains a design choice, not an independently demonstrated source of the overall gain. Table 2 (p. 8) and Appendix D.6 (p. 22) retain the control and state this attribution limit.
Response to Reviewer Sjwe (3)
Question 4 — Geometry–repair association and failure case
Following your suggestion, we tested the association using 40 archived paired queries. With k=5 fixed, both selected sets are evaluated under one common quality/kernel definition. Per-query differences in log-det and binary success are each residualized against mean-query-relevance differences. Partial Pearson r=.0944, with pointwise 95% query-pair bootstrap CI [−.1119,.3924] (200,000 resamples; seed20261110). Six outcome pairs are discordant, and the interval leaves the average association uncertain.
Geometry improves on all 40 queries, but both methods repair 4/40 (3 wins/3 losses). For Codeforces 926_I, IT-CES improves common log-det from −24.203898 to −21.088844 and lowers redundancy, yet omits the alarm's occupied minute, returning 24:00 instead of 23:59; Top-K's program is correct. Thus better geometry does not guarantee repair. The paired repair-rate p=1 is not a test of the partial correlation. Appendix E.1 (p. 23) gives the analysis, programs' behavior and separate Qwen3.8/CodeBERT development protocol; it is not pooled with the main evaluation.
Question 5 — Independent tests and model-generated errors
The new 800-problem cohort removes target-model failure filtering. It contains 280 HackerEarth, 240 AIZU and 280 Desc2Code problems, excludes groups used in retrieval, validation, the original test and prior development sets, and requires each buggy program to fail a diagnostic test and its reference to pass both test partitions. The KITE gains are 6/7/17 repairs across the three sources (descriptive, not separately significant; Table 12, p. 22). Thus the controlled gain extends beyond the original filtered cohort.
The existing generator-transfer check (Table 3, p. 9) uses the same original 170 problems at k=3. IT-CES versus the best baseline gives 42/35 repairs for Qwen2.5-Coder-14B/CodeBERT, 78/70 for Qwen3-Coder-Plus/CodeBERT and 47/41 for DeepSeek-Coder-33B/UniXcoder. These are descriptive improvements across generator–encoder configurations; they retain the original cohort and baseline settings, so they are distinct from the independent 800-problem, equally tuned comparison and do not remove the original selection bias.
The supported scope is repair of historical buggy programs. The abstract, Section 3.1 (p. 4), Figure 1 (p. 3) and conclusion (p. 10) now state this explicitly. Errors initially generated by each evaluated model remain untested; later-round failed model repairs do not constitute that initial-error cohort. Appendix E.2 (p. 23) retains IT-CES/Top-K/KITE=57/60/59 on 75 HumanEvalFix/QuixBugs problems (Qwen3.5:9b/UniXcoder, 57-record corpus, different verification, no established equal-trial tuning) and 6/10/6 on 30 queries (quantized Qwen3-Coder-Next, syntax recovery). These non-leading results qualify generalization: performance varies across protocols, and the simultaneous changes prevent isolating domain/provenance effects. They remain separate from the 800-problem comparison, whose result supports single-attempt repair of historical competitive-programming errors under the specified protocol.
Question 6 — Dynamic strong baselines
The available multi-round evidence supports dynamic IT-CES over static IT-CES and dynamic Top-K; the requested dynamic MMR/KITE comparison remains unevaluated. Static IT-CES yields 49→55→58 repairs, dynamic IT-CES 49→62→71 and dynamic Top-K 35→44→49. These observations favor diagnostic updating in the tested setting; extending the claim to dynamic MMR/KITE requires those controls. Exact prompt-token equality is not established, and Appendix C.1 (pp. 16–17) withdraws the unsupported token estimate. We have removed multi-round evaluation from the abstract’s contribution enumeration (p. 1). Appendix D.4/Table 10 (p. 21) retains the trajectories as secondary observations, and the conclusion (p. 10) states the missing dynamic baselines. The main controlled result concerns single-attempt selection.
Dear Reviewer,
Thank you again for your careful reading and thoughtful feedback. As the rebuttal deadline is approaching, we would like to kindly follow up and ask whether our response has adequately addressed your concerns.
If so, we would be very grateful if you could consider updating your rating accordingly. Otherwise, we would greatly appreciate any further feedback on remaining concerns and would be happy to provide additional clarification.
Thank you again for your time and consideration.
Response to Reviewer Sjwe (1)
Thank you for the detailed questions about novelty and generalization. They helped us clarify the repair-specific contribution and add an independent, equally tuned evaluation. We address each question below with the corresponding evidence and manuscript changes.
Weakness (unnumbered) — Novelty and breadth of evidence
The contribution is a concrete failure-conditioned repair selector and evidence of its effectiveness beyond the original cohort. The ingredients are established; the repair-specific construction and controlled comparison define our contribution. The design question is how to select complementary repair transformations from the problem, buggy code and diagnostic when the query has no known correct program. Its diagnostic support/quality construction configures a DPP without a target fix or generation during selection. “Information-theoretic” identifies an exact property of the chosen criterion: at fixed k, log-det is Gaussian entropy up to scale and an additive constant; it does not claim a new entropy identity or a repair theorem. Rewritten Section 3.2 (pp. 4–5) states this exact correspondence separately from repair utility. Section 2 (pp. 3–4) and Appendix A.2 (p. 13) identify the construction as a particular DPP, equivalent to an identically configured selector, and distinguish its geometric interpretation from an accuracy guarantee.
On 800 independent problems without target-model failure filtering, IT-CES adds 30 repairs over equally tuned KITE at the same five-example/one-output budget (Holm p=.048297; Table 2, p. 8), with 3.4 ms/query reranking cost (Appendix C.3, pp. 18–19). KITE shares the diagnostic inputs and calibrated support and already includes log-det, so this comparison tests the specific objective beyond those shared ingredients. MMR and fixed-support differences remain uncertain. Our claim is this complete-selector result; explaining it through independently beneficial components is a distinct, unresolved attribution question. Question 5 reports both positive generator-transfer checks and the generalization limits.
Question 1 — Calibrated Top-K discrepancy
Thank you for pointing out this inconsistency. Your equivalence argument is correct, and we apologize for the erroneous C1 comparison. Sort by descending raw cosine with candidate-ID ties. Ordinary Top-K returns
a[:k]; calibrated Top-K returnsa[:max(m,k)][:k], where m is the shortest .8-probability prefix. Both select identical ordered IDs, and the shared renderer produces identical prompts.The reported check confirms equality for all 170 problems at both budgets. C1 reuses Top-K outputs (35/170 and 32/170); we withdraw 34/170 and the unequal diagnostics. Individual ID/prompt traces are not included, so we do not claim to reconstruct the earlier summary error's computational cause. C1 is no longer independent evidence for calibration; the other paired comparisons do not use its withdrawn entries. Section 3.3 (pp. 5–6), Appendix C.3 (pp. 18–19) and Tables 4/8 (pp. 9/19) state the identity and corrections.