acceptodds
Under review as a conference paper at ICLR 2027

Information-Theoretic Corrective Example Selection (IT-CES): Calibrated In-Context Example Selection for Automated Code Correction

Abstract

In corrective in-context learning for automated code repair, the effectiveness of Large Language Model (LLM) self-correction depends substantially on the quality of retrieved demonstrations. Standard similarity-based selection methods, such as Top-K retrieval, often select examples independently and ignore correlations among candidates, leading to redundant contexts with diminishing marginal utility. We propose Information-Theoretic Corrective Example Selection (IT-CES), a theory-motivated framework for selecting a compact set of corrective demonstrations that jointly cover complementary repair patterns for the current failure. IT-CES represents each correction instance using a diagnostic tuple that combines the problem, the buggy solution, and the observed failure signal, and applies a calibrated log-determinant selection objective to choose examples that are both relevant to the current failure and complementary to one another. The calibration step mitigates the concentration of dense-retrieval similarity scores, improving the contrast of the diagnostic kernel used by the log-determinant objective. Across three code-repair benchmarks, IT-CES consistently improves Pass@1 repair accuracy over representative similarity-based and diversity-aware retrieval methods across the evaluated context budgets, generators, and dense retrievers. We further report step-by-step component ablations, paired exact statistical tests, intrinsic retrieval diagnostics, and multi-round dynamic repair evaluations. These results support the effectiveness of calibrated diagnostic example selection within the tested repair settings. The implementation is uploaded as supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, In-Context Learning, Bayesian Methods, Information Theory, Automated Code Correction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SjweOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the redundancy of similarity-based corrective demonstrations in LLM code repair, which can limit the complementary repair guidance provided by retrieved examples. It proposes IT-CES, a training-free example selection method that combines the problem statement, buggy code, and execution failure signal into a diagnostic query to retrieve relevant historical repair cases. It then applies distributional calibration to construct a candidate support set and greedily selects complementary demonstrations using a log-determinant objective that balances relevance and diversity. On a fixed hard set of 170 problems from three data sources, IT-CES achieves first-attempt repair success rates of 28.8% and 28.2% with three and five demonstrations, respectively, outperforming the strongest retrieval baselines in the main comparison by 4.7 and 5.9 percentage points. Ablation studies, evaluations across different model configurations, and multi-round dynamic repair experiments further support its effectiveness within the tested settings.

Critical strengths

The paper’s core strength lies in conditioning corrective example selection on the specific failure state of the current code while jointly considering relevance and complementarity at the set level. By combining the problem statement, buggy code, and execution failure signal, IT-CES retrieves targeted historical cases and uses calibration and log-determinant selection to reduce redundancy, allowing a limited context budget to cover more repair cues. The method requires no additional model training, incurs low reranking overhead, and achieves consistent improvements across the three evaluated data sources, different model configurations, and multi-round repair settings, demonstrating practical value.

Critical weaknesses

The paper’s core weakness is that it does not sufficiently establish its substantive novelty over existing example selection methods. The algorithm primarily combines diagnostic retrieval, candidate filtering, relevance weighting, and an existing log-determinant selection framework. The information-theoretic and Gaussian-process analyses provide design motivation but do not establish a theoretical connection between the selection objective and code repair success. Consequently, the contribution relies primarily on empirical improvements, while the evaluations remain concentrated on the same set of 170 difficult problems, leaving the applicability and robustness of these improvements in broader settings insufficiently established.

Questions for authors
  1. Why does calibrated Top-K produce different results?According to the method description, standardization and softmax preserve the similarity ranking, while support filtering retains a prefix of that ranking containing at least kk candidates. Calibrated Top-K should therefore select the same examples as ordinary Top-K. Why does Table 3 report an increase from 32 to 34 successful repairs? Could you provide the selected example IDs, their ordering, and the final prompts for both configurations to explain this difference?
  2. What is the substantive contribution beyond existing log-determinant selection methods? When using identical diagnostic representations, candidate pools, and hyperparameter tuning budgets, what distinguishes IT-CES from KITE or a standard quality–diversity DPP selector? Could you more clearly isolate the contributions of diagnostic representations, candidate support calibration, and quality weighting?
  3. Does distributional calibration help primarily by reducing the candidate pool? Could you add a control that retains the top MM candidates by raw similarity, with the candidate count matched to IT-CES’s support size for each query? This would help determine whether the gains arise from adaptive probability-based filtering or simply from excluding less relevant candidates.
  4. What direct evidence connects geometric coverage to repair effectiveness? The information-theoretic and Gaussian-process analyses primarily provide design motivation. Could you examine whether log-determinant gains correlate with repair success after controlling for relevance and demonstration count, and analyze failure cases where geometric complementarity does not translate into useful repair guidance?
  5. Do the results generalize to independent test sets and model-generated errors? The current 170-problem cohort is selected based on Qwen3-Coder-Next’s zero-shot failures, while the repair targets are fixed historical buggy submissions. Could you evaluate on an independent test set without this model-specific filtering, as well as on erroneous code generated by each evaluated model, to assess how test-set selection and error provenance affect the conclusions?
  6. Does dynamic multi-round repair outperform equally dynamic strong baselines? The multi-round experiments primarily compare Dynamic Top-K with Dynamic IT-CES. Could you include MMR and KITE baselines that also retrieve and select new demonstrations after every failed round, while controlling model calls and context token budgets? This would help distinguish the contribution of dynamic updating from that of IT-CES’s selection strategy.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Sjwe (1)

Thank you for the detailed questions about novelty and generalization. They helped us clarify the repair-specific contribution and add an independent, equally tuned evaluation. We address each question below with the corresponding evidence and manuscript changes.

Weakness (unnumbered) — Novelty and breadth of evidence

The contribution is a concrete failure-conditioned repair selector and evidence of its effectiveness beyond the original cohort. The ingredients are established; the repair-specific construction and controlled comparison define our contribution. The design question is how to select complementary repair transformations from the problem, buggy code and diagnostic when the query has no known correct program. Its diagnostic support/quality construction configures a DPP without a target fix or generation during selection. “Information-theoretic” identifies an exact property of the chosen criterion: at fixed k, log-det is Gaussian entropy up to scale and an additive constant; it does not claim a new entropy identity or a repair theorem. Rewritten Section 3.2 (pp. 4–5) states this exact correspondence separately from repair utility. Section 2 (pp. 3–4) and Appendix A.2 (p. 13) identify the construction as a particular DPP, equivalent to an identically configured selector, and distinguish its geometric interpretation from an accuracy guarantee.

On 800 independent problems without target-model failure filtering, IT-CES adds 30 repairs over equally tuned KITE at the same five-example/one-output budget (Holm p=.048297; Table 2, p. 8), with 3.4 ms/query reranking cost (Appendix C.3, pp. 18–19). KITE shares the diagnostic inputs and calibrated support and already includes log-det, so this comparison tests the specific objective beyond those shared ingredients. MMR and fixed-support differences remain uncertain. Our claim is this complete-selector result; explaining it through independently beneficial components is a distinct, unresolved attribution question. Question 5 reports both positive generator-transfer checks and the generalization limits.

Question 1 — Calibrated Top-K discrepancy

Thank you for pointing out this inconsistency. Your equivalence argument is correct, and we apologize for the erroneous C1 comparison. Sort by descending raw cosine with candidate-ID ties. Ordinary Top-K returns a[:k]; calibrated Top-K returns a[:max(m,k)][:k], where m is the shortest .8-probability prefix. Both select identical ordered IDs, and the shared renderer produces identical prompts.

The reported check confirms equality for all 170 problems at both budgets. C1 reuses Top-K outputs (35/170 and 32/170); we withdraw 34/170 and the unequal diagnostics. Individual ID/prompt traces are not included, so we do not claim to reconstruct the earlier summary error's computational cause. C1 is no longer independent evidence for calibration; the other paired comparisons do not use its withdrawn entries. Section 3.3 (pp. 5–6), Appendix C.3 (pp. 18–19) and Tables 4/8 (pp. 9/19) state the identity and corrections.

AuthorsOpenReview ↗

Response to Reviewer Sjwe (2)

Question 2 — Contribution and controlled comparisons

IT-CES specifies a failure-conditioned quality–diversity DPP and tests it against existing selectors under shared inputs and tuning budgets. It selects from the observable failure tuple without the query's correct program or generator calls. Score-shape-dependent support, raw-cosine quality weights and a diagnostic Gram kernel define the rule. KITE additionally optimizes query-prediction error; MMR uses pairwise redundancy. These are distinct preferences: with selected vectors e1,e2 and equal-relevance candidates (e1+e2)/sqrt(2) and .8e1+.6e3, MMR prefers the first (maximum cosine .7071<.8), whereas our shifted Gram kernel gives conditional residual variances .0286<.1867 and log-det prefers the second. This zero-jitter example explains the subset preference; repair outcomes provide the empirical test below. Section 3.2 (pp. 4–5) and Appendix A.2 (p. 13) state the geometric criterion and give its marginal, including jitter.

The component interventions are explicit: diagnostic/code-only Top-K repairs 32/26 problems; DPP configurations (alpha,p)=(1,1),(1,.8),(2,1),(2,.8) repair 36/40/43/48. Alpha=1 retains quality weighting. These are descriptive results, not independently established component effects (Table 4, p. 9; Appendix D.2, pp. 20–21).

Calibrated MMR/KITE share IT-CES's diagnostic embeddings, top-200 retrieval, p=.8 support, generator, prompt and k=5. IT-CES/MMR/KITE each receive 18 trials on the same separate 60 validation problems, with all 54 results disclosed in Table 6 (p. 18). This matches trial counts, not parameter spaces. On 800 independent problems:

MethodRepairs/800IT-only / comparator-onlyHolm p
IT-CES457——
Calibrated MMR42996 / 68.069361
Calibrated KITE42788 / 58.048297
Fixed top-42 IT-CES43887 / 68.147998

All three exact McNemar tests belong to one Holm family (Tables 2/13, pp. 8/22). KITE compares complete objectives; fixed42 changes support size alone. KITE already includes log-det. With diagnostic inputs and support held fixed, the gain supports the specified quality–diversity objective relative to KITE’s query-reconstruction objective, not the mere presence of diversity. MMR changes quality handling and subset scoring together, so it cannot identify log-det’s separate contribution. Fixed42 isolates support formation, with inconclusive evidence for an adaptive-support advantage. Thus the complete-objective comparison and component attribution answer different questions; the latter remains unresolved, rather than establishing zero component effects. “Calibrated” names the support operation, not an independently verified gain. Table 1 (p. 8) now displays original and stronger shared-support 170-problem results together; the controlled tests remain nonsignificant (Table 11, p. 22). All 800 paired outcomes are disclosed in Appendix F (p. 24).

Question 3 — Matched candidate count

Your proposed matched-count control is exactly equivalent: raw top-M with M equal to each query's nucleus size gives exactly the same support. Calibration changes cardinality, not ranking (Section 3.3, pp. 5–6). We therefore compare fixed top-42, retaining IT-CES's quality weights, kernel and greedy rule while changing only support formation.

Fixed42 repairs 438/800 versus IT-CES's 457/800: +2.375 percentage points, pointwise 95% CI [−.625,5.375], Holm p=.147998. This control isolates support formation while keeping the objective fixed. The interval includes zero and the corrected test is nonsignificant, establishing neither superiority nor equivalence. Adaptive support therefore remains a design choice, not an independently demonstrated source of the overall gain. Table 2 (p. 8) and Appendix D.6 (p. 22) retain the control and state this attribution limit.

AuthorsOpenReview ↗

Response to Reviewer Sjwe (3)

Question 4 — Geometry–repair association and failure case

Following your suggestion, we tested the association using 40 archived paired queries. With k=5 fixed, both selected sets are evaluated under one common quality/kernel definition. Per-query differences in log-det and binary success are each residualized against mean-query-relevance differences. Partial Pearson r=.0944, with pointwise 95% query-pair bootstrap CI [−.1119,.3924] (200,000 resamples; seed20261110). Six outcome pairs are discordant, and the interval leaves the average association uncertain.

Geometry improves on all 40 queries, but both methods repair 4/40 (3 wins/3 losses). For Codeforces 926_I, IT-CES improves common log-det from −24.203898 to −21.088844 and lowers redundancy, yet omits the alarm's occupied minute, returning 24:00 instead of 23:59; Top-K's program is correct. Thus better geometry does not guarantee repair. The paired repair-rate p=1 is not a test of the partial correlation. Appendix E.1 (p. 23) gives the analysis, programs' behavior and separate Qwen3.8/CodeBERT development protocol; it is not pooled with the main evaluation.

Question 5 — Independent tests and model-generated errors

The new 800-problem cohort removes target-model failure filtering. It contains 280 HackerEarth, 240 AIZU and 280 Desc2Code problems, excludes groups used in retrieval, validation, the original test and prior development sets, and requires each buggy program to fail a diagnostic test and its reference to pass both test partitions. The KITE gains are 6/7/17 repairs across the three sources (descriptive, not separately significant; Table 12, p. 22). Thus the controlled gain extends beyond the original filtered cohort.

The existing generator-transfer check (Table 3, p. 9) uses the same original 170 problems at k=3. IT-CES versus the best baseline gives 42/35 repairs for Qwen2.5-Coder-14B/CodeBERT, 78/70 for Qwen3-Coder-Plus/CodeBERT and 47/41 for DeepSeek-Coder-33B/UniXcoder. These are descriptive improvements across generator–encoder configurations; they retain the original cohort and baseline settings, so they are distinct from the independent 800-problem, equally tuned comparison and do not remove the original selection bias.

The supported scope is repair of historical buggy programs. The abstract, Section 3.1 (p. 4), Figure 1 (p. 3) and conclusion (p. 10) now state this explicitly. Errors initially generated by each evaluated model remain untested; later-round failed model repairs do not constitute that initial-error cohort. Appendix E.2 (p. 23) retains IT-CES/Top-K/KITE=57/60/59 on 75 HumanEvalFix/QuixBugs problems (Qwen3.5:9b/UniXcoder, 57-record corpus, different verification, no established equal-trial tuning) and 6/10/6 on 30 queries (quantized Qwen3-Coder-Next, syntax recovery). These non-leading results qualify generalization: performance varies across protocols, and the simultaneous changes prevent isolating domain/provenance effects. They remain separate from the 800-problem comparison, whose result supports single-attempt repair of historical competitive-programming errors under the specified protocol.

Question 6 — Dynamic strong baselines

The available multi-round evidence supports dynamic IT-CES over static IT-CES and dynamic Top-K; the requested dynamic MMR/KITE comparison remains unevaluated. Static IT-CES yields 49→55→58 repairs, dynamic IT-CES 49→62→71 and dynamic Top-K 35→44→49. These observations favor diagnostic updating in the tested setting; extending the claim to dynamic MMR/KITE requires those controls. Exact prompt-token equality is not established, and Appendix C.1 (pp. 16–17) withdraws the unsupported token estimate. We have removed multi-round evaluation from the abstract’s contribution enumeration (p. 1). Appendix D.4/Table 10 (p. 21) retains the trajectories as secondary observations, and the conclusion (p. 10) states the missing dynamic baselines. The main controlled result concerns single-attempt selection.

Dear Reviewer,

Thank you again for your careful reading and thoughtful feedback. As the rebuttal deadline is approaching, we would like to kindly follow up and ask whether our response has adequately addressed your concerns.

If so, we would be very grateful if you could consider updating your rating accordingly. Otherwise, we would greatly appreciate any further feedback on remaining concerns and would be happy to provide additional clarification.

Thank you again for your time and consideration.

Reviewer_UQv2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses how to select demonstrations for corrective in-context learning in automated code repair. The authors argue that standard similarity-based retrieval chooses examples independently, often filling the prompt with near-duplicate repair patterns that add little diagnostic information. They propose IT-CES, a training-free selector that represents each repair case as a diagnostic tuple combining the problem, buggy code, and observed failure signal. Motivated by an information-theoretic, uncertainty-reduction view of in-context learning, IT-CES selects examples using a calibrated quality–diversity log-determinant objective. This objective rewards candidates that are relevant to the current failure and complementary to one another. A calibration step counteracts the tendency of dense-retrieval similarity scores to cluster tightly, restoring contrast before greedy subset selection. On hard repair cases from three programming benchmarks, the method outperforms similarity-based, lexical, and diversity-aware baselines across context budgets and model configurations. Ablations attribute the gains to the diagnostic query and calibration components. The authors limit their conclusions to the tested settings.

Critical strengths
  1. Clear problem framing: Conditioning retrieval on the problem, buggy code, and failure signal directly targets the diagnostic redundancy that similarity-based selectors overlook in corrective prompts.
  2. Careful evaluation: Shared pools, step-by-step ablations, symmetric tuning, and paired exact tests with Holm correction.
  3. Practical and transparent: The selector is training-free and adds only milliseconds of reranking latency per query. Gains hold across generators, encoders, and benchmarks, and the authors frame the information-theoretic reasoning purely as design motivation rather than claiming formal performance guarantees.
Critical weaknesses
  1. Limited statistical scale: With 170 problems and single greedy runs, margins over the best baselines are eight to ten problems, and k=3 MMR and KITE comparisons barely pass Holm correction.
  2. Loose theoretical link: The information-gain and GP motivation offers no formal connection to repair accuracy.
  3. Reporting inconsistencies: The stated alpha search grid omits 1.5 and 2.5, yet sensitivity results report both values, undermining reproducibility claims.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer UQv2 (1)

Thank you for the precise comments on statistical scale, the theory–accuracy link and parameter reporting. We address each with the added evaluation, an explicit empirical check and a corrected account of parameter selection.

Weakness 1 — Statistical scale and uncertainty

We added 800 distinct problems without target-model failure filtering: 280 HackerEarth, 240 AIZU and 280 Desc2Code, excluding problem groups in retrieval, validation, the original 170 and prior development sets. Each method receives five examples and one generated candidate. IT-CES, calibrated MMR and calibrated KITE each use 18 validation trials on the same separate 60 problems, then freeze their parameters (Table 6, p. 18).

IT-CES repairs 457/800 (57.125%). The complete controlled panel is:

ComparatorRepairs/800Gain (percentage points)Pointwise 95% CIHolm p
Calibrated MMR429+3.500[.375,6.625].069361
Calibrated KITE427+3.750[.875,6.750].048297
Fixed top-42 IT-CES438+2.375[−.625,5.375].147998

The three two-sided exact McNemar tests form one Holm family. Source-stratified paired-bootstrap intervals are pointwise, not multiplicity-adjusted (200,000 resamples; seed20261110). The KITE comparison survives Holm correction: 30 additional repairs at the same five-example/one-output budget. MMR and fixed-support differences do not pass the corrected threshold. The gain is positive in all three sources (6/7/17 repairs; descriptive, not separate significance claims). The practical outcome is 30 additional correct programs without additional generator calls. Its p=.048297 remains close to the threshold; we therefore report the effect size, interval and complete family together. The MMR and fixed-support advantages remain uncertain: nonsignificance establishes neither zero effect nor equivalence. These two unresolved comparisons limit the breadth of the result, rather than reversing the KITE comparison. Tables 2/13 (pp. 8/22) report these results; Appendix F (p. 24) discloses all paired outcomes.

The original k=3 MMR/KITE comparisons remain marginal (corrected Holm p=.044922; Table 9, p. 21). The primary controlled comparison is the new panel: it removes failure filtering and equalizes tuning. Table 1 (p. 8) now shows the original and shared-support 170-problem panels together, with their distinct configurations and the latter’s nonsignificant tests stated in its caption (Table 11, p. 22). Different cohorts and baseline configurations preclude interpreting their p-value differences as a sample-size effect or pooling their evidence. Single greedy outputs measure variation across problems, not stochastic-decoding reliability; Section 4.2 (pp. 7–8) and Appendix D.1 (p. 20) state this scope.

Weakness 2 — Geometry versus repair accuracy

The objective supplies a geometric selection criterion; repair effectiveness is supported by the controlled outcome comparisons. We agree that these are distinct claims. Revised Section 3.2 (pp. 4–5) presents the implemented kernel and its precise interpretation: for fixed k and covariance L_S, Gaussian entropy is one-half log-det(L_S) plus a constant. This identity motivates geometric coverage. It neither models the LLM as that Gaussian nor gives a repair-accuracy bound. Appendix A.1 (p. 12) now explicitly labels its posterior interpretation as a modeling assumption. Appendix A.2 (p. 13) gives the relevance–volume decomposition and distinguishes fixed-cardinality log-det(L) from monotone log-det(I+L), whose approximation guarantee does not transfer. The abstract and conclusion (pp. 1/10) state the empirical scope.

The separate 40-query development check makes this boundary concrete: geometry improves on every query, yet IT-CES and Top-K each repair 4/40. With k=5 fixed and relevance differences controlled, partial Pearson r=.0944 (95% bootstrap CI [−.1119,.3924]); this neither establishes a positive association nor proves its absence. For Codeforces 926_I, improved geometry accompanies a repair that omits the alarm's occupied minute, producing 24:00 instead of 23:59. Appendix E.1 (p. 23) reports the analysis and failure; the paired repair-rate p=1 is not a correlation test.

AuthorsOpenReview ↗

Response to Reviewer UQv2 (2)

Weakness 3 — Validation grid versus sensitivity analysis

Thank you for catching this reporting ambiguity. The 1.5/2.5 values were post-selection test sensitivity checks, not validation candidates. The validation grid is alpha={1,2,3} × p={.50,.60,.70,.80,.90,.95}; its 18 trials select (2,.8), scoring 18/60. The five additional (alpha,p) points—(1.5,.75), (1.5,.8), (2,.75), (2,.85), (2.5,.8)—were evaluated afterward on the original test set, yielding 46/47/47/47/46 repairs, without selecting parameters.

Appendix C.2 and Table 6 (pp. 17–18) now separate the two procedures. Figure 2 (p. 9) labels the panel “Post-selection test sensitivity, N = 170 Hard Set”; its caption explains that these points are not the validation grid.

Dear Reviewer,

Thank you again for your careful reading and thoughtful feedback. As the rebuttal deadline is approaching, we would like to kindly follow up and ask whether our response has adequately addressed your concerns.

If so, we would be very grateful if you could consider updating your rating accordingly. Otherwise, we would greatly appreciate any further feedback on remaining concerns and would be happy to provide additional clarification.

Thank you again for your time and consideration.

Reviewer_2NMfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Information-Theoretic Corrective Example Selection (IT-CES), a method for selecting in-context examples for automated code correction. It constructs diagnostic tuples consisting of problem descriptions, buggy code, and observed execution failures, and uses a calibrated candidate selection pipeline followed by a quality-weighted log-determinant objective to select a diverse set of corrective examples. The authors evaluate IT-CES on 170 held-out repair problems and report improvements over several retrieval-based baselines. Additional experiments investigate component ablations, encoder transferability, and multi-round repair.

While the problem is practically relevant and the proposed framework combines several potentially useful ideas, I have substantial concerns about the consistency of the calibration procedure and the reproducibility of the experimental setup. In particular, the reported differences between Top-K and calibrated Top-K are difficult to reconcile with the monotonicity properties of the stated algorithm. These issues undermine the attribution of the reported improvements to the proposed components.

Critical strengths
  1. Motivated problem formulation. The paper identifies an important limitation of conventional retrieval-based prompting: examples selected based only on problem descriptions may not be relevant to the specific failure encountered during code correction. Incorporating buggy code and execution feedback into diagnostic tuples is a reasonable direction.

  2. Broad empirical evaluation. The authors investigate multiple retrieval baselines, component ablations, encoder transfer configurations, and multi-round repair. The use of paired comparisons and discordant-pair statistics also provides some transparency for evaluating the reported improvements.

Critical weaknesses
  1. Inconsistency in the calibration mechanism and insufficient evidence for the log-determinant contribution. This is my primary concern. According to Equations (2) and (3), the calibration transformation preserves the ranking of cosine similarity scores, and the nucleus filtering retains a prefix containing at least k candidates. Therefore, calibrated Top-K should select the same top-k examples as the original Top-K, assuming identical tie-breaking and no additional selection operation. However, Table 3 reports different repair results (32/170 vs. 34/170), while Table 6 also shows differences in redundancy and noise statistics. These discrepancies suggest that the actual implementation is inconsistent with the described procedure. More broadly, the current ablations do not sufficiently isolate the contribution of the log-determinant objective from calibration. The authors should clarify the exact selection procedure and provide stronger paired comparisons against calibrated MMR and validation-tuned KITE.

  2. Insufficient experimental reproducibility and dataset provenance. Appendix C.7 reports running the 80B-parameter Qwen3-Coder-Next model on a single 24GB RTX 4090, but does not specify the quantization, offloading, or serving configuration needed to reproduce this setup. In addition, the relationship between Desc2Code and its cited CodeSearchNet source, as well as potential overlap among the three data sources, remains unclear. The authors should provide complete model deployment details, dataset-to-source mappings, global problem identifiers, and cross-source deduplication procedures. These omissions make the experimental results difficult to independently verify.

  3. Insufficient novelty and positioning against prior information-theoretic ICL selection methods. The proposed list-wise information-theoretic example selection paradigm substantially overlaps with prior work, particularly Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translation (IEEE TSE 2025) and KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning. CAST also formulates exemplar selection as an information-coverage maximization problem and uses greedy submodular optimization to select complementary demonstrations, while KITE explores kernelized information-theoretic selection for ICL. Although IT-CES targets code correction and adopts diagnostic embeddings and a log-determinant objective, the underlying motivation of information-aware set-level ICL selection is closely related. The manuscript does not adequately discuss these connections or establish the methodological novelty of its approach. The authors should explicitly position IT-CES against these prior methods and clarify what fundamentally distinguishes its contribution.

Questions for authors
  1. Calibration mechanism and selection objective: Could the authors provide the exact pseudocode and selected candidate IDs for Top-K and calibrated Top-K? Given the monotonicity of the stated transformations, what explains the different selected-set diagnostics reported in Tables 3 and 6? Additionally, could the authors provide paired comparisons against calibrated MMR and validation-tuned KITE to better isolate the contribution of the log-determinant objective?

  2. Experimental reproducibility and dataset independence: Could the authors provide the exact Qwen3-Coder-Next model revision, weight precision, quantization, GPU/host-memory allocation, and offloading or serving configuration used in the experiments? How was the 80B-parameter model deployed on a single 24GB RTX 4090? Furthermore, could they clarify the original sources and splits of all datasets, provide global problem identifiers and cross-source deduplication rules, and specify the execution environments used for retrieval-time feedback and final hidden-test evaluation?

  3. Novelty and statistical validation: How does IT-CES fundamentally differ from prior information-theoretic ICL selection methods, such as CAST in Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translation and KITE (Kernelized and Information Theoretic Exemplars for In-Context Learning)? Beyond adapting existing selection principles to code correction, what is the fundamental methodological novelty of the proposed calibration and log-determinant objective? Could the authors strengthen these comparisons and correct the Holm-adjusted p-values in Table 7?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2NMf (1)

Thank you for identifying the calibration inconsistency and requesting stronger controlled comparisons. We have corrected the comparison and added an equally tuned evaluation on 800 independent problems.

Weakness 1 — Calibration consistency and paired controls

You are correct: calibrated Top-K is identical to Top-K. We apologize for the erroneous comparison. We withdraw 34/170 and the unequal diagnostics. With identical ordering and prompts, C1 reuses Top-K's outputs: 35/170 at k=3 and 32/170 at k=5. It provides no independent evidence for calibration. Neither the other-selector paired tests nor the added 800-problem comparison uses the withdrawn entries. Section 3.3 (pp. 5–6) now consistently distinguishes support calibration from raw-cosine quality weighting. Tables 4/8 (pp. 9/19) correct C1; Table 8 labels unreported diagnostics explicitly.

Question 1 gives the equally tuned 800-problem comparisons and distinguishes evidence for the complete rule from component attribution.

Weakness 2 — Deployment and dataset provenance

Main generation ran on an NVIDIA RTX PRO 5000 72GB Blackwell, rather than the originally reported 24GB RTX 4090. We apologize for the hardware misattribution. The archived local deployment uses Q4_K_M weights, whose 51.74GB model file fits this device; GPU monitoring records 56,919MiB usage. Appendices C.1/C.7 (pp. 16–17/20) correct the deployment account and withdraw the dependent GPU-hour estimate.

HackerEarth and AIZU contain historical Python submissions. Desc2Code is our ≥350-word-description subset of CodeContests train, not CodeSearchNet. All use project problem-group splits. Deduplication combines normalized statement hashes with five-word-shingle Jaccard≥.90; entire groups stay in one split. Recorded HE–AOJ/HE–Desc2Code/AOJ–Desc2Code overlaps are 4/12/7 groups, resolved in HE→AOJ→Desc2Code priority. Reported retrieval/validation/test group intersections are zero. Appendix B (pp. 14–16) gives these procedures; Question 2 below distinguishes the retained records from the missing replay metadata.

Weakness 3 — Contribution relative to CAST and KITE

IT-CES contributes a failure-conditioned repair selector: it specifies how to construct and select complementary demonstrations when the query’s correct program is unknown. The contribution is this repair-specific formulation and its controlled evaluation, building on established subset-selection principles. CAST maximizes AST-subtree coverage for translation; KITE combines query-prediction-error reduction with log-det diversity. IT-CES uses (problem, buggy code, diagnostic) for both query and candidate representations, restricts support using the local score distribution, and weights joint diagnostic volume by raw-cosine relevance. Demonstration fixes enter the prompt but not the selection kernel, matching an unfixed query's observables. Selection needs no generator calls; reranking costs 3.4 ms/query (Appendix C.3, pp. 18–19).

For nonsingular G_SS, the zero-jitter marginal is 2 log q_j + log(G_jj − G_jS G_SS⁻¹ G_Sj): relevance plus joint residual volume, versus MMR's maximum pairwise penalty. For a concrete distinction, take selected unit vectors e1,e2 and equal-relevance candidates a=(e1+e2)/sqrt(2), b=.8e1+.6e3. MMR prefers a (maximum cosine .7071 versus .8); our shifted Gram kernel gives zero-jitter residual variances .0286 versus .1867, so log-det prefers b. This illustrates the difference between joint and pairwise coverage; the outcome comparison below evaluates repair effectiveness. IT-CES is a particular quality–diversity DPP, and an identically configured DPP is equivalent. Section 2 (pp. 3–4) credits CAST/KITE; revised Section 3.2 (pp. 4–5) now starts from the implemented kernel rather than a Bayesian-to-accuracy chain. Appendix A.2 (p. 13) derives the jittered marginal.

The controlled comparison tests whether this particular repair-selection objective adds value beyond diagnostic retrieval and an existing log-det selector. It does: IT-CES repairs 30 more programs than equally tuned KITE at the same five-example/one-output budget (Section 4.2, pp. 7–8). Diagnostic inputs and support are shared, and KITE already includes log-det; those ingredients alone therefore do not explain the difference. This is evidence for the specified complete objective. Establishing which component causes that difference is a separate question, left unresolved by the ablations. Section 3.2 makes this distinction explicit.

AuthorsOpenReview ↗

Response to Reviewer 2NMf (2)

Question 1 — Exact selection rule and paired evidence

ordered = sort(pool, key=(-raw_cosine, candidate_id))
pi = softmax(zscore(scores_in_ordered_order))
m = shortest prefix whose probability sum reaches 0.8
TopK = ordered[:k]
CalibratedTopK = ordered[:max(m,k)][:k]

Both return the same ordered IDs. The renderer omits selector names/scores, so prompts also coincide. The reported implementation check confirms equality for 170/170 problems at both budgets. Individual ID/prompt traces are not retained in the supplied material, so the earlier summary error’s computational cause remains undetermined. C1's k=5 redundancy/coverage/Hit@5/Noise are therefore Top-K's .934/1.68/41.8%/8.2% (Appendix C.3, pp. 18–19; Table 8, p. 19).

On 800 independent problems without target-model failure filtering, all methods share diagnostic embeddings, top-200 retrieval, generation and k=5. IT-CES, calibrated MMR and calibrated KITE each receive 18 trials on the same separate 60 validation problems; calibrated baselines share p=.8 support while retaining their own scoring rules (Table 6, p. 18).

ComparatorRepairs/800IT-only / comparator-onlyHolm p
Calibrated MMR42996 / 68.069361
Calibrated KITE42788 / 58.048297
Fixed top-42 IT-CES43887 / 68.147998

IT-CES repairs 457/800. This follows your suggestion to compare calibrated MMR and validation-tuned KITE directly. These are two-sided exact McNemar tests in one three-comparison Holm family. The KITE gain is +3.750 percentage points, with pointwise 95% paired-bootstrap CI [.875,6.750]; all outcomes are disclosed in Appendix F (p. 24). Tables 2/13 (pp. 8/22) retain all comparisons. With diagnostic inputs and support shared, the KITE result supports the complete quality–diversity objective beyond an advantage from those inputs alone. Calibrated MMR changes both quality handling and subset scoring, so it is not an isolated log-det ablation: its nonsignificant result establishes neither a separate log-det benefit nor its absence. Fixed42 changes support size alone and leaves adaptive-support superiority and equivalence unresolved. Table 1 (p. 8) now places the original and stronger shared-support 170-problem results together; all six controlled tests remain nonsignificant (Table 11, p. 22).

Question 2 — Reproduction settings and test isolation

The revised appendices specify deployment, decoding, prompt construction and test isolation. The corrected generation hardware is one RTX PRO 5000 72GB; local deployment records identify qwen3-coder-next:q4_K_M under Ollama. The author-reported main-cohort decoding used temperature=0, top_p=1, max_tokens=2048, repetition_penalty=1, seed=20260901 and one candidate. Empty, truncated, invalid, timed-out and incorrect outputs count as failures; completed model failures were not retried (Appendix C.1, pp. 16–17).

Appendix C.5 (pp. 19–20) replaces the abbreviated template with the complete shared renderer: description, buggy program, diagnostic and fixed program for each example, ordered by raw query similarity with candidate-ID ties. The shared 256/128/128 field budgets apply to embedding inputs across all sources; generation receives complete examples. Appendix C.1’s subheading now reflects this shared allocation. Appendix B.1 (p. 15) now describes long statements in terms of this truncation constraint, removing the unsupported claim of full-description embedding degradation. This disclosure also corrects the unsupported token estimate: methods share demonstration count and rendering, not exact prompt-token count.

The source snapshots are HE-20260801, AOJ-20260801 and D2C-20260801; the source and group-deduplication rules are given under Weakness 2. HE_/AO_/DC_ are internal record IDs, not public upstream IDs. After input deduplication, approximately 20% of tests provide diagnosis and 80% provide final acceptance. Acceptance inputs, answers and feedback never enter retrieval or prompting; reference programs pass both partitions (Appendix B, pp. 14–16).

The retriever is jinaai/jina-code-embeddings-1.5b, revision 39aeb4fb9b60f930934c78ae5d749a46287c248a, with last-token pooling, 1536 dimensions and L2 normalization (Appendix C.1, pp. 16–17). Appendix F (p. 24) discloses all paired outcomes for independent statistical recomputation. Complete regeneration remains limited by missing cohort-bound generator hashes/backend receipts, host-RAM allocation, exact checker OS/interpreter/resource versions and the upstream-ID crosswalk. The local deployment archive establishes feasibility; it does not independently verify each reported cohort request.

AuthorsOpenReview ↗

Response to Reviewer 2NMf (3)

Question 3 — Methodological distinction and Holm correction

The names identify the implemented criterion and support operation. “Information-theoretic” refers to the exact fixed-cardinality Gaussian-entropy interpretation of log-det in Section 3.2, not an estimated mutual information or repair theorem. “Calibrated” denotes distribution-standardized support sizing, not calibrated probabilities of correctness or a separately demonstrated gain. Positive affine score changes leave z-scores and support unchanged; calibration neither repairs absolute score compression nor changes Top-K ordering (Section 3.3, pp. 5–6). The failure-conditioned construction in Weakness 3 and its equally tuned comparison in Section 4.2 (pp. 7–8) define the contribution.

We corrected Holm using the cumulative-maximum rule. For Top-K/BM25/KITE/MMR, corrected p values are .010307/.022614/.044922/.044922 at k=3 and .003422/.013275/.015076/.015076 at k=5. These original-panel comparisons remain significant (Table 9 and Appendix D.1, pp. 20–21). The separate 800-problem family retains only KITE significance; no comparison is dropped.

Dear Reviewer,

Thank you again for your careful reading and thoughtful feedback. As the rebuttal deadline is approaching, we would like to kindly follow up and ask whether our response has adequately addressed your concerns.

If so, we would be very grateful if you could consider updating your rating accordingly. Otherwise, we would greatly appreciate any further feedback on remaining concerns and would be happy to provide additional clarification.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.