acceptodds
Under review as a conference paper at ICLR 2027

Beyond Retry: A Budget-Controlled Ablation of Compiler Feedback for Lean Proof Repair

Abstract

Formal theorem-proving agents frequently feed Lean compiler diagnostics back to language models after failed proof attempts. Yet existing improvements from compiler-guided repair do not establish which part of that feedback is responsible: semantic error information, a binary failure verdict, additional prompt context, or simply another opportunity to generate. We present a budget-controlled evaluation that isolates the informational value of compiler feedback in iterative Lean proof repair. Starting from 219 failed attempts of a public checkpoint (deepseek-ai/DeepSeek-Prover-V1.5-RL) on a version-pinned Lean 4 benchmark (LeanDojo Benchmark 4 (v10)), we compare four feedback conditions: full diagnostics, binary success/failure, no feedback, and length-matched non-diagnostic placebo text. All conditions use identical model checkpoints, theorem splits, decoding seeds, maximum generated tokens, prompt-length caps, and Lean verifier-call budgets; the primary comparisons are full diagnostics against the placebo and against the binary verdict, which differ from it only in the feedback text. At the pre-specified budget of 8 verifier calls and 3 seeds, the paired full-minus-placebo difference in solved rate is +0.9 percentage points (95% interval [-2.4, +4.0]) and the paired full-minus-binary difference is +1.2 ([-2.1, +4.4]); both intervals include zero. The secondary full-minus-none difference, which also changes the state context, is -9.9 ([-14.5, -5.3]): independent resampling from the same initial failure reached accepted proofs more often than any condition that showed the model its previous attempt. For this model and benchmark, the length-matched placebo and the binary verdict recover whatever the repair loop adds, and we find no evidence that diagnostic semantics add to retry and context effects at equal budgets. By checker-observable failure category, no bucket shows a primary difference whose interval excludes zero; bucket slices are descriptive. We release scripts, environment locks, prompts, seeds, raw per-attempt records and complete checker traces; one command regenerates every table and figure.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Lean, LLMs, proof repair, compiler feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qqyXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper isolates compiler-diagnostic information from binary failure feedback, length-matched placebo context, and independent retries for a fixed DeepSeek-Prover-V1.5-RL checkpoint. It starts with 219 failed LeanDojo attempts and holds verifier-call budgets, seeds, decoding, and prompt caps fixed. At eight calls and three seeds, full diagnostics improve solved rate by 0.9 percentage points over placebo (95% interval [-2.4, +4.0]) and 1.2 over binary feedback ([-2.1, +4.4]); independent resampling performs better by 9.9 points. Strict acceptance under the original theorem and checker auditing support a useful controlled negative result for this model and interface.

Critical strengths
  1. The ablation asks a well-defined causal question. The binary and length-matched placebo controls separate diagnostic semantics from a failure verdict, prompt length, and extra generation opportunity, under equal verifier-call budgets (Sections 3–4). These are stronger controls than a simple feedback-versus-no-feedback comparison.

  2. The strict acceptance checks and reported replay audit strengthen the credibility of the outcome measure: repaired proofs must establish the original theorem without sorry, added axioms, or self-lemma substitution, and the paper reports checking all 235 accepted proofs plus 600 failures. Paired intervals and explicitly descriptive failure-category slices are appropriate for the experiment.

Critical weaknesses
  1. The primary null comparisons do not establish equivalence. The intervals permit diagnostic gains of up to 4.0–4.4 percentage points, substantial relative to the roughly 6% placebo/binary solved rates. The claim that these controls 'recover whatever the repair loop adds' is therefore stronger than the evidence. The supported conclusion is that this experiment does not detect an additional diagnostic benefit at the tested budget, with the reported bounds; an equivalence claim would require a justified margin and corresponding analysis.

  2. The experiment has limited ability to separate lack of diagnostic value from an interface/checkpoint unable to use diagnostics. It uses one checkpoint, with the previous proof and feedback in Lean block comments under a completion prompt, rather than validating an explicitly instructed repair interface. The much higher copying rates in contextual conditions and the strong independent-retry result make this distinction material. A shared Lean-version mismatch also need not affect conditions equally, because diagnostic text is the treatment. The narrow result is useful, but does not yet explain why feedback fails or establish a broader limitation of compiler-guided repair.

Questions for authors
  1. Please replace equivalence wording with the bounded null result, or specify a meaningful equivalence margin and show the corresponding analysis. What diagnostic benefit does the experiment rule out?

  2. Is there a positive control demonstrating that this checkpoint and exact prompting interface can respond correctly to an actionable diagnostic? Would an explicit repair instruction, with the same budgets and conditions, change copying rates or the primary comparison?

  3. Can a diagnostic-capable repair checkpoint or a version-matched run distinguish interface/toolchain effects from informational redundancy? This would materially strengthen the interpretation rather than simply broaden a model leaderboard.

Reviewer_nWEKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposed an analysis to focus on if Lean compiler diagnostics actually provides additional repair information, rather than just benefit from retry. They use the same model (DeepSeek-Prover-V1.5-RL) and let it retry with 4 conditions (FULL, BINARY, PLACEBO and NONE) to retry and see if it's fixed, and how complete compiler diagnostics didnt bring any extra advantage statistically.

Critical strengths

Overall this paper tried to discover a potentially useful question, and the critical strengths are:

  1. Complete story (though in a limited experimental setup): The experiment setup is relatively complete, including the FULL, BINARY, PLACEBO setup and the maintained experimental condition along the way, showing certain level of rigorousness. This wont change the fact that this work is still kind of below the bar of acceptance.

  2. One interesting negative finding: In Table 2 it reveals that NONE achieved 16.9% solved rate, which is much higher than FULL 7.0%, and it's actually about the much higher repeated candidate from FULL compared with NONE, and it shows than showing model failed proof can make the search diversity worse. This strength wont change the assessment to be positive, just an interesting angle.

Critical weaknesses

This work has many issues to prevent it from being accepted:

  1. Problematic experimental setup: The paper only used DeepSeek-Prover-V1.5-RL and this model doesn't have any context regarding compiler diagnostic, so the model choice itself can potentially change all the conclusion that the authors claimed. For example, FULL didn't gain advantage is simply because the model itself is not capable of repairing anything. This is critical and directly put the paper into rejection.

  2. NONE and PLACEBO control issues: In the experimental setup NONE removes not only diagnostics but also previous failed candidates, so it changed multiple factors. Similarly, PLACEBO didn't match the setup of FULL too. This question the experiment rigorousness and directly put the paper into rejection bucket.

Questions for authors
  1. Why choosing this model? Will any conclusion get changed if you choose different, potentially more capable model, or models specifically fine-tuned in this scenario?

  2. Can you explain the NONE and PLACEBO setup, and potentially report results with corrected setup?

Reviewer_gt1QOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the informational value of compiler feedback in iterative Lean proof repair. It conducts a controlled ablation study comparing 4 conditions: FULL, PLACEBO, BINARY and NONE. The results show that FULL does not significantly outperform PLACEBO or BINARY, while performing substantially worse than NONE.

Critical strengths

The effectiveness of compiler feedback in proof repair is an important question worth investigating. The paper demonstrates that directly providing full compiler feedback does not necessarily lead to better performance under the paper’s evaluated setting, offering potentially useful insights.

The paper carefully controls experimental variables and pays much attention to technical details.

Critical weaknesses
  1. The depth of analysis is insufficient in this paper. The paper mainly finds through experiments that feedback does not improve the proof success rate under the evaluated setting, but does not sufficiently investigate the underlying reasons. I believe that, for a paper primarily focused on empirical analysis, the experiments are not sufficiently comprehensive. For example, the paper does not provide a finer-grained analysis of which parts of the feedback can be effectively utilized by the model, or how such information affects the proof repair process. The paper also lacks experiments with multiple models to investigate whether the observed results are due to the evaluated model's inability to understand and utilize feedback, or other possible factors.

  2. The paper contains too many engineering details, making the content overly verbose and reducing its information density. As a result, the main contributions are not sufficiently highlighted, and the paper is not very smooth to read. In addition, the discussions of figures, tables and experimental results mostly restate the observed results without sufficiently analyzing the underlying reasons.

Questions for authors
  1. Could the authors provide a trajectory-level analysis comparing the specific repair behaviors of the agent under FULL, BINARY, and PLACEBO? For example, does the model use diagnostic information to correct specific errors from the previous attempt, even when the final proof remains unsuccessful?

  2. Is there a causal relationship between the candidate repetition rate and the final proof success rate? Could the authors provide further experimental evidence to determine whether higher candidate repetition rates actually cause lower proof success rates, rather than merely being correlated with them?

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.