- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper isolates compiler-diagnostic information from binary failure feedback, length-matched placebo context, and independent retries for a fixed DeepSeek-Prover-V1.5-RL checkpoint. It starts with 219 failed LeanDojo attempts and holds verifier-call budgets, seeds, decoding, and prompt caps fixed. At eight calls and three seeds, full diagnostics improve solved rate by 0.9 percentage points over placebo (95% interval [-2.4, +4.0]) and 1.2 over binary feedback ([-2.1, +4.4]); independent resampling performs better by 9.9 points. Strict acceptance under the original theorem and checker auditing support a useful controlled negative result for this model and interface.
Critical strengths
-
The ablation asks a well-defined causal question. The binary and length-matched placebo controls separate diagnostic semantics from a failure verdict, prompt length, and extra generation opportunity, under equal verifier-call budgets (Sections 3–4). These are stronger controls than a simple feedback-versus-no-feedback comparison.
-
The strict acceptance checks and reported replay audit strengthen the credibility of the outcome measure: repaired proofs must establish the original theorem without sorry, added axioms, or self-lemma substitution, and the paper reports checking all 235 accepted proofs plus 600 failures. Paired intervals and explicitly descriptive failure-category slices are appropriate for the experiment.
Critical weaknesses
-
The primary null comparisons do not establish equivalence. The intervals permit diagnostic gains of up to 4.0–4.4 percentage points, substantial relative to the roughly 6% placebo/binary solved rates. The claim that these controls 'recover whatever the repair loop adds' is therefore stronger than the evidence. The supported conclusion is that this experiment does not detect an additional diagnostic benefit at the tested budget, with the reported bounds; an equivalence claim would require a justified margin and corresponding analysis.
-
The experiment has limited ability to separate lack of diagnostic value from an interface/checkpoint unable to use diagnostics. It uses one checkpoint, with the previous proof and feedback in Lean block comments under a completion prompt, rather than validating an explicitly instructed repair interface. The much higher copying rates in contextual conditions and the strong independent-retry result make this distinction material. A shared Lean-version mismatch also need not affect conditions equally, because diagnostic text is the treatment. The narrow result is useful, but does not yet explain why feedback fails or establish a broader limitation of compiler-guided repair.
Questions for authors
-
Please replace equivalence wording with the bounded null result, or specify a meaningful equivalence margin and show the corresponding analysis. What diagnostic benefit does the experiment rule out?
-
Is there a positive control demonstrating that this checkpoint and exact prompting interface can respond correctly to an actionable diagnostic? Would an explicit repair instruction, with the same budgets and conditions, change copying rates or the primary comparison?
-
Can a diagnostic-capable repair checkpoint or a version-matched run distinguish interface/toolchain effects from informational redundancy? This would materially strengthen the interpretation rather than simply broaden a model leaderboard.
Dear authors and reviewers,
The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.
AC