- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper distinguishes a counterexample's formal version-space informativeness from the repair behavior it induces in an LLM. Across Datalog, regex, and list-DSL tasks, it measures exactness, safe uptake, progress, regression, and damage, finding weak or reversed associations between symbolic informativeness and repair quality. TEACHCE predicts these learner-specific consequences and combines them into an acquisition score; TEACHCE-AR uses limited adaptive feedback while retaining generated repairs. Controlled, multi-model, and higher-replication experiments support the mismatch finding, while public-program experiments test transfer. The decision-theoretic results characterize ideal expected utility and prediction-error bounds rather than guaranteeing optimality of the implemented router.
Critical strengths
-
The central mismatch is supported by a substantial controlled empirical investigation. Executable evaluation across three domains, multiple learners, prompt perturbations, alternative symbolic scores, and a higher-replication audit make the weak relationship between formal information and actual repair more persuasive than a single noisy ranking experiment (Sections 4–5; Appendices C–D). This is a useful distinction for feedback design.
-
The selector is evaluated with meaningful controls: observable-only deployment is distinguished from target-relative diagnostic headroom, and ordering comparisons hold the adaptive executor fixed. On the fresh 120-task ordering experiment, TEACHCE-AR reaches 39.2% exact repair versus 29.2% for maximum symbolic informativeness, with 1.67 versus 1.77 calls (Table 12). These results support a practical case for modeling the learner rather than relying only on symbolic elimination.
Critical weaknesses
-
The public-program evidence establishes a narrower advantage than general superiority of learned ordering. The three frozen static comparisons in Table 16 have confidence intervals spanning zero; in Table 17, static TEACHCE achieves 56.3% full-suite correctness versus 60.3% for maximum information. Adaptive TEACHCE→shortest reaches 66.8%, close to information→TEACHCE at 66.3%, with 1.34 versus 1.26 calls. The main public-program comparison of 90.4% versus 89.6% also needs its population size and paired interval. Adaptive reuse is promising, but learned ordering's additional transfer benefit remains uncertain. 'Exact' on these programs should mean benchmark-test correctness unless semantic equivalence is established.
-
Central selector details are not sufficiently reproducible from the manuscript: the exact observable feature vector, tree hyperparameters, grouped cross-validation/search settings, and construction of protected/regression sets need an executable specification (Appendix A.4). This matters because the contribution depends on these learned consequences. The joint score also has tradeoffs: in the fresh ordering test, catastrophe rate is 14.2%, versus 10.8% for shortest feedback, so the quality/safety improvement should not be described as uniform.
Questions for authors
-
Please supply the exact observable feature contract, tree and cross-validation configuration, protected/regression-set definitions, and frozen selector. Are these available in a reproducible artifact?
-
What are the paired uncertainty intervals for the decisive sequential and public-program comparisons, including Table 5's population size and its relationship to the 199-program replay? At matched total calls/tokens, how much improvement is attributable to learned ordering beyond the shared adaptive executor?
-
Does validating the consequence heads or varying the catastrophe/damage weights explain the higher catastrophe rate in the fresh ordering experiment? This would affect the interpretation of the joint acquisition objective.
Response to Reviewer BV8w [Part I]
Thank you for your thoughtful review, particularly for highlighting the controlled mismatch investigation and the importance of holding the adaptive executor fixed in the fresh ordering comparison. We will address your questions one by one below.
1. W2 / Q1 — Executable Selector Specification and Reproducibility
The complete executable specification is detailed below, and the full codebase, feature extractors, training records, and frozen selector bundle will be open-sourced upon acceptance.
[Paper Revision] We will expand Appendix A.4 with the full feature-contract table, exact tree/CV hyperparameters, tie-breaking rules, and preservation-set definitions, and release the complete reproducible artifact publicly upon acceptance.
2. W1 / Q2 — Public Populations, Paired Uncertainty, and Matched-Compute Ordering Gains
At matched total calls and strict per-task token ceilings, learned counterexample ordering delivers substantial exact-recovery gains (+9.17 to +11.67 pp) over symbolic and shortest baselines.
At the 3,000-token ceiling, TEACHCE outperforms shortest by +9.17 pp and maximum information by +11.67 pp.
[Paper Revision] We will clarify the Table 5 population size (25 programs), define "Exact" explicitly as full benchmark-suite pass on public C programs, and add the strict token-ceiling table to Appendix G.
3. W2 / Q3 — Catastrophe Calibration and Final-Return Preservation Filter
The catastrophe head is well-calibrated (Brier 0.059); the 17/120 catastrophes occur because the executor's fallback rule can return a non-preserving candidate when no rollout meets early-stop criteria. Adding a hard final-return filter eliminates all observed catastrophes (17/120 → 0/120) while preserving 39.2% exact recovery.
Tracing all 17/120 catastrophic final returns confirms that 100% of them violate the known preservation suite (Rt). Enforcing a hard final-return preservation filter reduces observed catastrophes from 17/120 (14.2%) to 0/120 (0.0%) while retaining 39.2% exact recovery and 48.3% safe uptake with zero additional LLM or verifier calls.