acceptodds
Under review as a conference paper at ICLR 2027

When Counterexamples Don't Teach: Learner-Specific Feedback in LLM Repair

Abstract

Counterexamples are widely used to guide program synthesis and repair: a verifier exposes a concrete failure, and the learner revises its current solution from that feedback. Existing selection strategies largely judge counterexamples by how informative they are about the hypothesis space. This is natural for symbolic learners, but less obvious when the updater is an LLM, whose response to the same valid feedback can vary substantially. We show that formal informativeness is often a poor predictor of whether an LLM will make a useful repair. The mismatch appears across several LLMs and executable task families. Motivated by this observation, we introduce TEACHCE, which selects counterexamples according to the repair consequences they are expected to induce in the current learner, and an adaptive verifier-guided variant that reuses repair attempts when needed. Across controlled repair tasks and sequential settings, this learner-specific view improves the quality–compute trade-off over symbolic selection. On public student-program benchmarks, we further show that combining consequence-aware selection with adaptive verifier-guided rollout reuse improves repair efficiency under a limited inference budget. These results suggest that, once LLMs enter the repair loop, choosing feedback requires modeling not only what the evidence says, but also how the learner will respond to it.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Large Language Models, Counterexample-guided Inductive Synthesis, Program Repair

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BV8wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper distinguishes a counterexample's formal version-space informativeness from the repair behavior it induces in an LLM. Across Datalog, regex, and list-DSL tasks, it measures exactness, safe uptake, progress, regression, and damage, finding weak or reversed associations between symbolic informativeness and repair quality. TEACHCE predicts these learner-specific consequences and combines them into an acquisition score; TEACHCE-AR uses limited adaptive feedback while retaining generated repairs. Controlled, multi-model, and higher-replication experiments support the mismatch finding, while public-program experiments test transfer. The decision-theoretic results characterize ideal expected utility and prediction-error bounds rather than guaranteeing optimality of the implemented router.

Critical strengths
  1. The central mismatch is supported by a substantial controlled empirical investigation. Executable evaluation across three domains, multiple learners, prompt perturbations, alternative symbolic scores, and a higher-replication audit make the weak relationship between formal information and actual repair more persuasive than a single noisy ranking experiment (Sections 4–5; Appendices C–D). This is a useful distinction for feedback design.

  2. The selector is evaluated with meaningful controls: observable-only deployment is distinguished from target-relative diagnostic headroom, and ordering comparisons hold the adaptive executor fixed. On the fresh 120-task ordering experiment, TEACHCE-AR reaches 39.2% exact repair versus 29.2% for maximum symbolic informativeness, with 1.67 versus 1.77 calls (Table 12). These results support a practical case for modeling the learner rather than relying only on symbolic elimination.

Critical weaknesses
  1. The public-program evidence establishes a narrower advantage than general superiority of learned ordering. The three frozen static comparisons in Table 16 have confidence intervals spanning zero; in Table 17, static TEACHCE achieves 56.3% full-suite correctness versus 60.3% for maximum information. Adaptive TEACHCE→shortest reaches 66.8%, close to information→TEACHCE at 66.3%, with 1.34 versus 1.26 calls. The main public-program comparison of 90.4% versus 89.6% also needs its population size and paired interval. Adaptive reuse is promising, but learned ordering's additional transfer benefit remains uncertain. 'Exact' on these programs should mean benchmark-test correctness unless semantic equivalence is established.

  2. Central selector details are not sufficiently reproducible from the manuscript: the exact observable feature vector, tree hyperparameters, grouped cross-validation/search settings, and construction of protected/regression sets need an executable specification (Appendix A.4). This matters because the contribution depends on these learned consequences. The joint score also has tradeoffs: in the fresh ordering test, catastrophe rate is 14.2%, versus 10.8% for shortest feedback, so the quality/safety improvement should not be described as uniform.

Questions for authors
  1. Please supply the exact observable feature contract, tree and cross-validation configuration, protected/regression-set definitions, and frozen selector. Are these available in a reproducible artifact?

  2. What are the paired uncertainty intervals for the decisive sequential and public-program comparisons, including Table 5's population size and its relationship to the 199-program replay? At matched total calls/tokens, how much improvement is attributable to learned ordering beyond the shared adaptive executor?

  3. Does validating the consequence heads or varying the catastrophe/damage weights explain the higher catastrophe rate in the fresh ordering experiment? This would affect the interpretation of the joint acquisition objective.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BV8w [Part I]

Thank you for your thoughtful review, particularly for highlighting the controlled mismatch investigation and the importance of holding the adaptive executor fixed in the fresh ordering comparison. We will address your questions one by one below.


1. W2 / Q1 — Executable Selector Specification and Reproducibility

The complete executable specification is detailed below, and the full codebase, feature extractors, training records, and frozen selector bundle will be open-sourced upon acceptance.

  • Observable feature contract: The selector extracts 86 observable features for Datalog, 50 for Regex, and 46 for List DSL, encoding current-hypothesis syntax, labeled candidate counterexamples, symbolic statistics, and candidate-pool summaries. Automated invariance tests verify that all deployment features are invariant to removing hidden target programs and mutation metadata.
  • Model & hyperparameter configuration: In the primary controlled benchmark, each of the six consequence heads is an ExtraTreesRegressor with 400 trees, min_samples_leaf=3, max_features=1.0. Acquisition weights for (Zexact,G~,Zsafe,Zreg,Zcat,D)(Z_{\text{exact}}, \tilde{G}, Z_{\text{safe}}, Z_{\text{reg}}, Z_{\text{cat}}, D) are fixed at (1,0.25,0.5,−0.5,−2,−2)(1, 0.25, 0.5, -0.5, -2, -2) with wI=0w_I = 0 (and wI=0.1w_I = 0.1 in the legacy public-program Ridge selector with regularization 100100). For the paired two-backbone comparison, development-only 5-fold task-grouped CV searches over ExtraTrees, RandomForest, GradientBoosting, and Ridge.
  • Protected / regression sets & neighborhoods: Controlled preservation sets (RtR_t) contain 32 inputs on which hth_t and h∗h^* agree (class-balanced when possible). Structural neighborhoods (N(x)\mathcal{N}(x)) contain up to 8 domain-generated related failures used strictly for evaluation and excluded from selector features. Sequential prompts display 8 preservation examples, while acceptance checks the maintained suite.
    [Paper Revision] We will expand Appendix A.4 with the full feature-contract table, exact tree/CV hyperparameters, tie-breaking rules, and preservation-set definitions, and release the complete reproducible artifact publicly upon acceptance.

2. W1 / Q2 — Public Populations, Paired Uncertainty, and Matched-Compute Ordering Gains

At matched total calls and strict per-task token ceilings, learned counterexample ordering delivers substantial exact-recovery gains (+9.17 to +11.67 pp) over symbolic and shortest baselines.

  • Public-program populations & definitions:
  1. Table 5 population: Evaluates 25 C-Pack programs from one exercise (5 repetitions each, 125 trajectories), completely disjoint from the 199-program replay (Table 17). On these public C benchmarks, "Exact" denotes full benchmark test-suite pass (as noted in Table 17), rather than formal semantic equivalence.
  2. Paired uncertainty: In the 256-task sequential benchmark (Table 3), Sequential TEACHCE improves over maximum information by +17.19 pp at S@1 and +4.69 pp at S@5.
  • Isolating learned ordering at matched calls & strict token budgets (120 tasks):
  1. Matched calls: Before rollout reuse can contribute (1 call), TEACHCE improves exact recovery by +11.67 pp over maximum information (34.2% vs. 22.5%) and +10.00 pp over shortest (34.2% vs. 24.2%). At a cap of 2 calls, TEACHCE achieves 39.2% exact recovery using only 1.792 total API attempts and 1,051 tokens/task (including retries).
  2. Strict per-task token ceilings: Using a conservative admission rule that reserves prompt bytes, framing overhead, and the 800-token output cap before each request (returning hth_t if the budget is exhausted): |Per-task token ceiling (cap ≤2 repairs)|TEACHCE|Shortest CE|Max Symbolic Info|Structural Utility| |---|---|---|---|---| |2,000 tokens|18.3%|13.3%|10.8%|11.7%| |3,000 tokens|37.5%|28.3%|25.8%|35.8%| |5,000 tokens|39.2%|31.7%|28.3%|36.7%|

At the 3,000-token ceiling, TEACHCE outperforms shortest by +9.17 pp and maximum information by +11.67 pp.

[Paper Revision] We will clarify the Table 5 population size (25 programs), define "Exact" explicitly as full benchmark-suite pass on public C programs, and add the strict token-ceiling table to Appendix G.


3. W2 / Q3 — Catastrophe Calibration and Final-Return Preservation Filter

The catastrophe head is well-calibrated (Brier 0.059); the 17/120 catastrophes occur because the executor's fallback rule can return a non-preserving candidate when no rollout meets early-stop criteria. Adding a hard final-return filter eliminates all observed catastrophes (17/120 →\to 0/120) while preserving 39.2% exact recovery.

Tracing all 17/120 catastrophic final returns confirms that 100% of them violate the known preservation suite (RtR_t). Enforcing a hard final-return preservation filter reduces observed catastrophes from 17/120 (14.2%) to 0/120 (0.0%) while retaining 39.2% exact recovery and 48.3% safe uptake with zero additional LLM or verifier calls.

Reviewer_eiBWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies counterexample selection for LLM-based program repair. The authors argue that classical counterexample selection criteria, such as symbolic informativeness, do not necessarily reflect the usefulness of feedback for LLM-based learners. Unlike symbolic learners, LLMs may respond to an informative counterexample with a local correction, an ineffective revision, or a change that introduces regressions.

Motivated by this observation, the authors propose shifting from informativeness to teachability, which measures the expected semantic improvement induced by a counterexample for a particular learner. They introduce TEACHCE, a learned counterexample-selection method using six lightweight consequence predictors, and TEACHCE-AR, which additionally incorporates verifier-guided adaptive rollout reuse.

Critical strengths
  • Empirical identification of a mismatch between counterexample informativeness and LLM repair effectiveness

The paper provides an interesting empirical observation that formal informativeness is weakly correlated with the actual effectiveness of counterexamples in LLM-based program repair. The authors further distinguish between local correction, structural repair, regression, and exact recovery, demonstrating that fixing an observed counterexample does not necessarily imply meaningful semantic improvement.

The analysis across multiple controlled domains and LLMs, together with executable verification of repair outcomes, provides useful evidence for this distinction and motivates reconsidering conventional counterexample-selection strategies for LLM-based learners.

Critical weaknesses
  • Limited methodological novelty and questionable generalizability of the learned selector.

While the motivation is interesting, proposed methodological contribution is relatively limited. TEACHCE essentially trains six lightweight predictors on previously observed LLM repair outcomes and combines their predictions. The core approach appears to be a relatively straightforward application of supervised outcome prediction and heuristic selection, rather than a fundamentally new mechanism for improving LLM-based repair.

More importantly, unlike conventional training-free, learner-agnostic counterexample-selection methods, the proposed approach requires additional learner-specific supervision.

The training data also primarily comes from the same controlled task families used for evaluation. Although the authors appropriately separate development and evaluation tasks, this does not establish whether the learned predictors generalize beyond familiar repair distributions. It remains unclear whether TEACHCE captures a generalizable notion of teachability or primarily learns learner- and domain-specific repair patterns.

  • Limited practical gains and insufficient evidence of the learned selector's effectiveness

The paper identifies an interesting and potentially important mismatch between formal counterexample informativeness and actual LLM repair effectiveness. While the proposed method improves repair outcomes in controlled settings, I find the practical improvements relatively limited compared with the additional learner-specific supervision.

In particular, Section 5 provides evidence that consequence-aware selection improves repair performance. However, the gains are considerably smaller when compared with stronger, simpler baselines or when evaluated on realistic program-repair benchmarks. For example, Table 5 reports only a 0.8 percentage-point improvement in exact repair accuracy over shortest-counterexample and maximum-information selection, both of which are training-free baselines.

Additionally, the paper's central motivation concerns the weak association between symbolic informativeness and realized repair quality, as illustrated in Figure 1. Although the downstream improvements provide evidence supporting the proposed approach, a directly comparable analysis of the learned acquisition score's correlation with empirical repair utility would further substantiate the claimed mechanism.

Questions for authors

Could the authors further clarify how the proposed method addresses the mismatch between symbolic informativeness and actual LLM repair effectiveness highlighted in Figure 1? While paper demonstrates improvements in downstream repair performance, I would appreciate further clarification on how effectively TEACHCE solves the mismatch of symbolic informativeness and actual LLM repair effectiveness.

Additionally, are there particular settings or scenarios where the proposed approach offers more substantial practical advantages over simpler, training-free methods? I may be overlooking some of the intended contributions or practical benefits, and would appreciate the authors' perspective.

2 comments
AuthorsOpenReview ↗

Response to Reviewer eiBW [Part I]

Thank you for recognizing the value of distinguishing local correction, structural repair, regression, and exact recovery. We appreciate your suggestions to connect our learned acquisition score directly to empirical utility and to clarify the practical regimes where TEACHCE excels. Below, we provide direct mechanism alignment evidence, structural/decomposition controls, and matched-budget comparisons, alongside planned revisions to improve clarity.


1. W1 — Methodological Contribution and Generalization

TEACHCE formulates counterexample selection as a decision-theoretic intervention problem, outperforming both a structural repair-difficulty baseline (+8.33 pp at one call) and a single-head direct-utility predictor (+3.33 pp).

  • Conceptual & algorithmic formulation: Classical selection rules rank evidence by hypothesis-space elimination (Et(x)E_t(x)), which assumes a consistent symbolic updater. Once an LLM enters the loop (ht+1∼qθ(⋅∣ht,x,y∗,f)h_{t+1} \sim q_\theta(\cdot \mid h_t, x, y^*, f)), selection becomes choosing an intervention. Rather than heuristics, TEACHCE decomposes expected repair consequences into six complementary signals—exact recovery (ZexactZ_{\text{exact}}), safe uptake (ZsafeZ_{\text{safe}}), semantic progress (G~\tilde{G}), regression risk (ZregZ_{\text{reg}}), catastrophic failure (ZcatZ_{\text{cat}}), and downside magnitude (DD)—combined via a plug-in acquisition rule A^w(x)=∑jwjm^j(x)+wIE~t(x)\hat{A}_w(x) = \sum_j w_j \hat{m}_j(x) + w_I \tilde{E}_t(x) with formal one-step and finite-horizon regret guarantees (Theorems 2, 6, and 8). This multi-head decomposition allows practitioners to inspect risk vs. reward separately and adjust utility weights without recollecting repair data.
  • Controlled baselines isolating our contribution (120 unseen tasks): To verify that gains come from multi-head consequence modeling rather than general syntactic difficulty or feature engineering alone, we added two controlled baselines trained on the exact same development repairs: |Selector|Exact recovery (1 call)|Exact recovery (Adaptive cap = 2)| |---|---|---| |Maximum symbolic information|22.5%|29.2%| |Shortest CE|24.2%|34.2%| |Structural repair-difficulty predictor|25.8%|36.7%| |Single-head direct utility (full features)|—|35.8%| |TEACHCE (6-head consequence model)|34.2%|39.2%|
  • At one call (isolating pure ranking before rollout reuse), TEACHCE outperforms the structural difficulty predictor (using program size, syntax, and CE complexity) by +8.33 percentage points.
  • Genuine learner–CE interaction: Across 9,000 repairs (36 tasks ×\times 10 CEs ×\times 5 learner configurations ×\times 5 repetitions), removing both task-level learner means and shared CE difficulty reveals statistically significant, reproducible learner–CE interactions across all six consequence heads, confirming that feedback utility depends jointly on the learner rather than additive task difficulty alone.
  • Generalization without oracle supervision: Beyond unseen program pairs and cross-model transfer (Table 4 & Table 15), we show that TEACHCE can be trained using only 64 finite tests per task and with all symbolic/version-space features completely removed, demonstrating applicability where exhaustive semantic evaluation or version-space enumeration is unavailable.

[Paper Revision] We will add the structural difficulty and single-head controls to Section 5.2, and move the finite-test and symbolic-feature-free protocols to Section 4 / Appendix E to make the methodological assumptions and generalization scope clear.


2. W2 / Q1 — Direct Alignment Between Learned Acquisition Score and Empirical Repair Utility

Unlike symbolic informativeness, which exhibits near-zero or negative correlation with empirical utility, TEACHCE's learned acquisition score achieves a strong, statistically significant positive correlation across both joint utility and safe uptake.

Following your suggestion for a direct Figure 1-style mechanism comparison, we evaluated the within-task Spearman correlation of both symbolic informativeness (Et(x)E_t(x)) and our learned acquisition score (A^w(x)\hat{A}_w(x)) against empirical repair outcomes across 360 unseen counterexamples:

Selection CriterionCorr. with Empirical Joint UtilityCorr. with Empirical Safe Uptake
Symbolic Informativeness (Et(x)E_t(x))−0.024−0.086
TEACHCE Acquisition Score (A^w(x)\hat{A}_w(x))+0.203+0.157
Paired Correlation Gain+0.227+0.243

Whereas formal informativeness fails to rank useful interventions (−0.024 and −0.086), TEACHCE aligns significantly with realized repair utility (+0.227 improvement) and regression-free safe uptake (+0.243 improvement), directly explaining its downstream selection gains.

[Paper Revision] We will add this direct correlation table and discussion to Section 5.2 (referencing Figure 1) to explicitly close the loop between the empirical mismatch and our learned acquisition criterion.

AuthorsOpenReview ↗

Response to Reviewer eiBW [Part II]

3. W2 / Q2 — Practical Regimes Where TEACHCE Offers Substantial Advantages

TEACHCE provides its largest practical gains in four high-value settings: (1) strict single-call budgets, (2) hard token ceilings, (3) regression-sensitive tasks, and (4) amortized multi-query deployment.

While multi-step rollout reuse naturally narrows the gap between orderings when generous budgets allow fallback retries (as in Table 5, where TEACHCE-AR still achieves the highest progress at 84.0% vs. 80.8% and lowest token usage at 1.60k vs. 1.75k for CEGIS), TEACHCE delivers substantial improvements over training-free baselines whenever compute, latency, or safety is constrained:

Practical Regime / ConstraintTEACHCE / TEACHCE-ARShortest CEMax Symbolic InfoGain vs. Best Baseline
1-Call Budget (Exact recovery, n=120n=120)34.2%24.2%22.5%+10.00 pp
3,000-Token Hard Ceiling (Exact recovery, n=120n=120)37.5%28.3%25.8%+9.17 pp
Low-Effort Reasoning (Safe uptake, n=36n=36)50.0%28.7%—+21.30 pp
Adaptive Cap = 2 (Billed attempts / tokens, n=120n=120)1.79 / 1,051 tok1.93 / 1,184 tok—−11.2% tokens at higher accuracy
  1. Strict 1-call & hard token budgets: When only one LLM call is permitted before rollout reuse, TEACHCE improves exact recovery by +10.00 pp over shortest and +11.67 pp over maximum information. Under a strict 3,000-token ceiling per task (including retries and up to two repairs), TEACHCE outperforms shortest by +9.17 pp and maximum information by +11.67 pp.
  2. Regression-critical repair: Across frozen decoding and reasoning-effort audits, TEACHCE boosts safe uptake by +11.1 to +21.3 pp over shortest and cuts catastrophic regression rates roughly in half compared with maximum information. Combining TEACHCE-AR with a preservation filter eliminates observed catastrophes entirely (0/120 vs. 17/120 without filtering) while preserving 39.2% exact recovery.
  3. Low-data training & fast amortized inference: Training requires only a one-time offline collection and incurs zero LLM probe calls and only 3.4–5.4 ms median selection overhead at deployment (Table 6).

[Paper Revision] We will reorganize Section 6 to highlight these compute-constrained and safety-critical deployment regimes.

Reviewer_5s7JOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies counterexample selection in LLM-based program repair. The authors argue that conventional symbolic informativeness, such as version-space reduction and information gain, does not necessarily predict whether an LLM can use a counterexample to make a useful repair.

To address this issue, the paper introduces TeachCE, which uses supervised predictors to estimate the consequences of showing a counterexample to a particular LLM. These predictions include exact recovery, semantic progress and regression risk. TeachCE-AR further combines counterexample ranking with adaptive verifier-guided repair.

The paper evaluates the approach on controlled Datalog, regular-expression and list-program tasks, as well as C-Pack and IntroClass. The results show a weak relationship between formal informativeness and actual LLM repair quality. Learner-specific selection also improves several repair metrics in controlled settings.

I find the research question interesting and relevant. However, I have concerns about the generality of the proposed method, the supervision required to train TeachCE, and whether the empirical improvements on real programs are sufficiently strong to support the broader claims.

Critical strengths
  1. The paper identifies an interesting distinction between the formal informativeness of a counterexample and its actual usefulness to an LLM. This is a meaningful question for neuro-symbolic learning and program repair.
  2. The empirical analysis is relatively comprehensive. The authors study several program representations, different LLMs and multiple repair outcomes. The repeated-repair experiments are also useful for distinguishing genuine effects from stochastic variation.
  3. I appreciate that the paper does not measure repair quality only by whether the displayed counterexample is fixed. It also considers semantic progress, regression and recovery of the target behaviour.
  4. The comparison between matched and swapped selectors is interesting. It provides some evidence that the choice of feedback depends on the particular learner.
Critical weaknesses

1. Generalisation of the learner-specific consequence predictor is not sufficiently established

This is my main concern.

TeachCE learns a predictor of the expected repair consequences of showing a counterexample to an LLM. However, these consequences depend on the learner itself.

In Section 3, the repair distribution is formulated as

ht+1∼qθ(⋅∣ht,x,ht(x),h∗(x),f).h_{t+1} \sim q_\theta(\cdot \mid h_t, x, h_t(x), h^*(x), f).

This distribution depends on the model parameters, the current hypothesis, the counterexample and the feedback format. In practice, it may also be sensitive to the repair prompt, reasoning configuration and decoding strategy.

However, the consequence predictor is trained offline using repair outcomes collected under particular learner configurations. It is therefore unclear how well the predictor generalises when these configurations change.

The cross-swap experiment in Appendix G.1 is relevant. It shows that matched selectors perform better than swapped selectors. This supports the claim that feedback value is learner-dependent. However, it also suggests that the learned selection policy may not transfer well across models.

I would like to see a more systematic evaluation of this limitation. For example, the authors could train a selector using one LLM and one prompt, then evaluate the same frozen selector with different prompts, reasoning efforts and model versions.

It would also be useful to report the performance degradation under these changes, rather than only comparing separately trained selectors.

The paper should clarify whether TeachCE is intended as a model-specific component that must be retrained for each learner, or as a more general counterexample-selection method.

Both settings are reasonable. But their practical implications are very different.

2. The method relies on unusually strong semantic supervision in the controlled benchmarks

A second major concern is how the training labels for the consequence predictor are obtained.

In the controlled experiments, the authors can evaluate the semantic consequences of a repair over a complete finite universe. For example, Datalog is evaluated over 512 Boolean worlds, and Regex over 5,461 strings.

This makes it possible to compute quantities such as

Gt=Err⁡X(ht,h∗)−Err⁡X(ht+1,h∗).G_t = \operatorname{Err}_{\mathcal X}(h_t,h^*) - \operatorname{Err}_{\mathcal X}(h_{t+1},h^*).

However, such exact semantic evaluation is normally not available in realistic software repair problems.

For a general program, we usually have only a finite test suite, partial specifications or counterexamples generated by a verifier. We cannot necessarily compute the total number of incorrect inputs or determine whether a repair is semantically equivalent to the intended program.

This raises an important question about the applicability of TeachCE.

How can its consequence predictor be trained when exact recovery, global semantic progress and catastrophic regression cannot be measured?

I suggest an experiment where all consequence labels are constructed using only a limited test suite. The test suite should be separated from the evaluation tests. This would better reflect the supervision available in practical program repair.

The authors could then compare the resulting selector with the one trained using complete semantic information.

Such an experiment would directly test whether the proposed method remains effective under realistic supervision.

3. Formal informativeness is difficult to compute in large program spaces

The paper compares TeachCE against symbolic informativeness based on version-space splitting.

Equation (1) defines

Et(x)=min⁡(∣{h∈Vt:h(x)=0}∣,∣{h∈Vt:h(x)=1}∣).E_t(x)=\min\left(|\{h\in V_t:h(x)=0\}|,|\{h\in V_t:h(x)=1\}|\right).

This is well defined for a finite and explicitly represented hypothesis space. However, computing it may be infeasible for a large program synthesis space.

The controlled domains in this paper use restricted program languages and bounded evaluation universes. These conditions are useful for studying the phenomenon, but they are quite different from general software engineering tasks.

I am not arguing that counterexamples cannot be generated for complex programs. Testing, symbolic execution and SMT-based verification can generate useful counterexamples without enumerating the whole hypothesis space.

The concern is more specific: exact version-space informativeness and complete semantic repair quality may not be available in such settings.

The paper should clearly distinguish counterexample generation from counterexample informativeness estimation, and discuss how TeachCE operates when only approximate information scores are available.

This would help readers understand which assumptions are necessary for the proposed method and which are only used for experimental analysis.

4. Is TeachCE learning learner-specific behaviour, or mainly predicting repair difficulty?

The paper argues that feedback selection should depend on the learner's response.

I agree with this motivation. However, the reported analyses suggest that repair structure and input complexity explain much of the variation in repair success.

For example, Appendix D.5 shows that the required semantic transition can be more predictive of safe uptake than symbolic informativeness.

This raises another possibility: TeachCE may mainly learn which types of program modifications are easier, rather than a more detailed model of how a particular LLM responds to feedback.

These two explanations should be distinguished.

I suggest comparing TeachCE with a simpler repair-difficulty predictor using only observable structural features, such as program size, counterexample complexity and syntactic characteristics of the current hypothesis.

A carefully controlled feature ablation would also be useful. In particular, how much additional performance comes from learner-specific training, beyond general repair-difficulty features?

The cross-model experiments provide some evidence, but do not fully answer this question.

I think this distinction is important for assessing the actual methodological contribution.

5. Improvements on real program repair are less convincing than those on controlled tasks

The controlled experiments show meaningful gains from learner-specific selection. However, the results on real student programs are more mixed.

For example, Table 5 reports 90.4% exact repair for TeachCE-AR, compared with 89.6% for the CEGIS baseline on C-Pack. This is a rather small improvement.

More importantly, Appendix G.2 reports that none of the three blind comparisons between frozen TeachCE selection and information gain on C-Pack and IntroClass achieves a confidence interval excluding zero.

The paper acknowledges this limitation, which I appreciate.

Nevertheless, these results suggest that the main benefits of TeachCE may be more reliable in restricted, fully evaluable program spaces than in realistic repair tasks.

I encourage the authors to make this distinction clearer in the main discussion and conclusions.

The gains obtained through adaptive rollout reuse should also be separated from those obtained through learner-specific counterexample ranking. Both are useful, but they are different mechanisms.

Questions for authors
  1. How many LLM repair calls and verifier executions are needed to construct the training data for TeachCE? Please report the complete offline training cost, including consequence-label collection, rather than only the deployment overhead.
  2. Does a selector trained using one repair prompt generalise to another prompt for the same LLM? What happens when the reasoning effort or decoding temperature changes?
  3. Can TeachCE be trained using only incomplete test-suite feedback, without access to the complete semantic universe? How much does this affect prediction quality and final repair success?
  4. How should symbolic informativeness be estimated when the version space cannot be enumerated? Does TeachCE still provide an advantage when both its features and supervision are approximate?
  5. Would a simple structural repair-difficulty predictor achieve similar results? Please compare it with the multi-head consequence predictor under the same training data and inference budget.
  6. How much improvement is contributed by counterexample ranking itself, and how much by adaptive rollout reuse and the regression-preserving acceptance rule? A decomposition using the same verifier and call budget would help clarify this.
  7. Can the authors provide stronger evidence on real program repair, preferably across larger or more diverse programs and with held-out tests that were not used to train the consequence predictor?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5s7J [Part I]

Thank you for recognizing the distinction between formal informativeness and LLM teachability, and for your constructive feedback on supervision assumptions, configuration sensitivity, and empirical boundaries. We address each question with new controlled experiments and explicit manuscript revisions.


1. Q1 — Offline Training Cost and Data Efficiency

Collecting the 120-task training bundle required 23,834 API requests (13.3M tokens), while a 4× smaller training set (5,480 repairs) retains 36.7% exact recovery (+7.50 pp over maximum information, 95% CI [1.67, 14.17]).

  • Collection & labeling cost: The full bundle covers 22,560 repairs across 292 development tasks (21,534 parse-valid repairs labeled; 1,026 invalid outputs marked as failures) plus 1,274 formatting retries. Per-domain caching reduces program–input labeling evaluations by 95.9%:
DomainTraining RepairsUncached EvaluationsCached Executions
Datalog7,68016,251,624402,944
Regex7,440153,137,4166,542,278
List DSL7,44038,070,1001,509,690
Total22,560207,459,1408,454,912
  • Low-data & deployment efficiency: Training with 1 repair per CE (5,480 records) yields 36.7% exact recovery (vs. 39.2% full bundle; 29.2% max info). Offline fitting is a one-time amortized cost; deployment requires no LLM probes and adds 3.4–5.4 ms median CPU overhead per task (Table 6).

[Paper Revision] We will add this complete offline cost breakdown and the 1-repair-per-CE ablation to Section 5.2 and Appendix C.3.


2. W1 / Q2 — Transfer and Sensitivity Across Learner Configurations

A frozen selector retains strong safe-uptake advantages across prompt, temperature, and reasoning-effort shifts, while exact recovery is more sensitive to configuration changes.

TEACHCE's consequence-learning procedure is general, whereas a fitted predictor targets a specific learner configuration and should be revalidated upon major configuration shifts. Holding the source selector completely frozen across four transfer experiments:

  1. Cross-model transfer (36 tasks, 5 repairs/CE): Evaluating a frozen source selector across five distinct backends yields consistent bounded-progress gains over maximum symbolic information: +10.43 pp (deepseek-v4-flash\texttt{deepseek-v4-flash}), +13.01 pp (deepseek-v4-pro\texttt{deepseek-v4-pro}), +15.16 pp (gpt-5.6-sol\texttt{gpt-5.6-sol}), +17.75 pp (qwen3-235b-a22b-instruct-2507\texttt{qwen3-235b-a22b-instruct-2507}), and +15.63 pp (gemini-3.6-flash\texttt{gemini-3.6-flash}), with 95% CIs excluding zero for four of the five models.
  2. Prompt transfer (12 Datalog tasks): Frozen TEACHCE matches or beats maximum information on unseen prompts: 27.8% vs. 19.4% (Original), 16.7% vs. 16.7% (Minimal edit), and 16.7% vs. 5.6% (Verify then edit).
  3. Temperature & reasoning-effort transfer (1,728 new repairs on 36 tasks, DeepSeek V4 Flash, 4,096-token cap):
ConfigurationTEACHCE exactShortest exactMaximum information exactTEACHCE safe uptakeShortest safe uptake
Non-thinking, temperature 0.221.3%16.7%19.4%38.9%26.9%
Non-thinking, temperature 0.816.7%13.9%18.5%35.2%24.1%
Thinking, low effort23.1%16.7%24.1%50.0%28.7%
Thinking, high effort21.3%17.6%21.3%30.6%22.2%
  • Safe uptake & degradation: Frozen TEACHCE improves safe uptake over shortest by +11.1 to +21.3 pp across temp 0.2, temp 0.8, and low effort (all 95% CIs above zero) and roughly halves catastrophic regressions across all four settings (3.7%–9.3% vs. 10.2%–19.4% for max info).

[Paper Revision] Section 3 will formally define a learner configuration (model, prompt, decoding, reasoning effort, and output budget) and specify when revalidation is needed. The temperature/effort transfer study will be added to Appendix G.1.


3. W2 / Q3 — Training from Incomplete Test-Suite Feedback

Supervision from just 64 sampled tests per task yields 36.4% exact recovery (+7.22 pp over maximum information, 95% CI [1.39, 13.61]), closely approaching full-semantic supervision (39.2%).

We retrained all six consequence heads using only 16, 64, or 256 sampled tests per development task (plus the displayed CE when absent) and evaluated on 120 disjoint tasks over their complete ground-truth universes (3 sampling seeds):

Training supervisionExact recoverySafe uptake
16 tests plus displayed CE36.1%44.2%
64 tests plus displayed CE36.4%45.0%
256 tests plus displayed CE37.5%46.4%
Full-semantic labels39.2%48.3%

Across 1,200 evaluation candidates, Brier / AUC scores shift moderately from full-label to 64-test supervision, while preserving over 90% of the downstream recovery gain.

[Paper Revision] We will add this finite-test supervision study to Appendix E and clarify that operational deployment requires only executable test suites.

AuthorsOpenReview ↗

Response to Reviewer 5s7J [Part II]

4. W3 / Q4 — Operating Without Exact Version-Space Enumeration

TEACHCE works effectively using sampled hypothesis approximations—or with all symbolic-information features removed (37.5% exact recovery, +8.33 pp over maximum information, 95% CI [2.50, 14.72]).

Counterexample generation (via testing or solvers) is separate from scoring via exact version-space enumeration. Under 64-test finite supervision, we tested two practical regimes:

  1. Zero symbolic features (120 tasks across all three domains): Removing all version-space features achieves 37.5% exact recovery vs. 29.2% for maximum information.
  2. Sampled hypothesis approximation (40 Datalog tasks, 3 seeds): Estimating splitting and entropy from M∈{32,128}M \in \{32, 128\} rejection-sampled consistent hypotheses during both training and inference outperforms maximum sampled information by +15.00 pp (M=32M=32) and +16.67 pp (M=128M=128): | Sampled hypotheses | TEACHCE | Maximum Sampled Information | Shortest | |---|---:|---:|---:| | 32 | 41.7% | 26.7% | 40.0% | | 128 | 40.8% | 24.2% | 40.0%|

[Paper Revision] Sections 3 and 4 will clearly distinguish counterexample generation from informativeness scoring and specify both the sampled-approximation and symbolic-feature-free interfaces.


5. W4 / Q5 — Disentangling Learner-Specific Consequences from Structural Difficulty

Multi-head consequence prediction outperforms a structural repair-difficulty baseline by +8.33 pp at one call (34.2% vs. 25.8%), and variance decomposition confirms significant learner–CE interactions.

  1. Structural difficulty & single-head baselines (120 tasks): A structural predictor trained on program size, syntax, and CE complexity lags behind TEACHCE at one call (25.8% vs. 34.2%). Holding full features fixed, 6-head consequence prediction also outperforms single-head direct utility (39.2% vs. 35.8%): |Selector | Exact recovery, one call | Exact recovery, shared adaptive cap of two | |---|---:|---:| | Maximum information | 22.5% | 29.2% | | Shortest | 24.2% | 34.2% | | Structural utility | 25.8% | 36.7% | | TEACHCE | 34.2% | 39.2%|
  2. Reproducible learner–CE interaction (9,000 repairs: 36 tasks × 10 CEs × 5 learners × 5 repetitions): After removing task-level learner means and shared CE difficulty effects, cross-repetition splits confirm positive reproducible interactions across all six heads, ruling out a purely additive difficulty model.

[Paper Revision] We will add the results to Appendix E.3.


6. W5 / Q6 — Component Decomposition: Ranking, Reuse, and Regression Protection

Consequence ranking contributes +10.00 pp at one call, adaptive reuse adds +5.00 pp while cutting calls to 1.67, and the regression filter eliminates observed catastrophes (17/120 →\to 0/120).

Holding candidate pools, repair records, and the verifier fixed across the 120 evaluation tasks:

Method / Component AblationExact recoveryObserved catastrophesMean calls
Shortest CE (1 call)24.2%—1.00
TEACHCE ranking only (1 call)34.2%—1.00
Second-ranked CE only (1 call)21.7%—1.00
Fixed 2-call TEACHCE + verifier retention39.2%17 / 1202.00
TEACHCE-AR (early stopping)39.2%17 / 1201.67
TEACHCE-AR + hard preservation filter39.2%0 / 1201.67
  • Ranking: Single-call TEACHCE beats shortest by +10.00 pp (95% CI [3.33, 17.50]).
  • Regression protection: Removing the regression term increases preservation-suite regressions by +4.17 pp; adding a hard final-return filter (falling back to the original program on 50/120 tasks) preserves 39.2% recovery with 0/120 observed catastrophes.

7. W5 / Q7 — Larger Real Programs (Codeflaws) and Scope of Claims

An added evaluation on 59 real C programs from Codeflaws confirms the feasibility of finite-test training on larger programs, while showing no statistically significant accuracy gain over shortest selection.

We evaluated 59 real C programs from distinct Codeflaws problems, split into 30 development programs (324 repairs) and 29 confirmation programs assessed on 576 disjoint hidden tests (without exact version-space features). At a 2-repair adaptive budget, hidden-test correctness is 56.32% (TEACHCE), 57.47% (shortest), 57.47% (same-feature direct utility), and 54.02% (structural utility).

  • Scope of claims: Together with C-Pack and IntroClass, these results clarify where each mechanism contributes most: pure learner-specific ranking yields its strongest gains in structured, specification-driven synthesis and repair domains, whereas on larger real-world C programs, static ranking alone does not outperform simple heuristics, and practical gains come primarily from adaptive rollout reuse and regression-preserving verification.

[Paper Revision] We will add the Codeflaws study to Appendix G and Conclusions to explicitly state this boundary between controlled domains and larger real-world programs.

Thank you for the extensive additional experiments. However, my main concern remains the practical contribution of TeachCE. The new results does not outperform simple counterexample selection on realistic settings. Moreover, the method requires substantial offline repair data to train its consequence predictors. I therefore maintain my reservations about the overall contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.