- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies autoregressive modeling of serialized Lean kernel statements and proofs, comparing kernel terms, surface Lean, and model-generated English for the same Mathlib theorems. It develops a deterministic term serialization, trains models from scratch and transfers Qwen3 checkpoints to the new vocabulary, analyzes proof-loss concentration, and fits model/data scaling laws. It also checks sampled proofs with Lean and relates reference-proof likelihood to a lower bound on generation success. The main findings are highly concentrated kernel-term token loss, substantial transfer from pretrained weights, and steeper measured loss-reduction trends for kernel terms, although current surface-Lean generation is stronger.
Critical strengths
1. Useful empirical study. The comparison across three proof representations provides valuable insights.
2. Reliable experiments. Careful verification and baseline controls strengthen the evaluation.
Critical weaknesses
- Limited novelty. Learning from Lean proof terms is well established in PACT (ICLR 2022), and Palermo et al. (AITP 2022) already trained language models to generate proof terms and filtered outputs using the Lean kernel. More directly, Yeh and Ong (2026) investigated kernel-term predictability and preliminary scaling laws. Therefore, the paper does not sufficiently distinguish these contributions from the closest prior work or establish what substantially new insight they provide beyond extending an existing experimental framework.
- Unclear motivation and background. The introduction jumps from a vague question about learning a “minimal grammar” to technical details and numerical findings without establishing a clear research problem. It does not adequately explain how kernel-term modelling differs from generating Lean source or tactics, why this distinction matters, or what limitation of existing approaches it addresses. Terms such as “warm start” and “reference prefixes,” together with proof-loss statistics, appear before readers are given sufficient context to interpret them.
Moreover, Table 1 largely repackages the abstract into a text-heavy table. It mixes claimed contributions, numerical results, implementation details, and conditional extrapolations without clearly distinguishing their significance. Overall, this paperlooks like more of an experimental report rather than a scholarly paper.
- Unclear method formulation. Although the paper defines prediction losses and scaling fits, it lacks a clear, unified formulation of the overall pipeline. The relationships between theorem statements, proof representations, model inputs and outputs, and kernel verification are scattered across the main text and appendices. Training, teacher-forced evaluation, and free-running generation are also not clearly distinguished in a consolidated overview. A precise task definition and an algorithm or pipeline diagram would help readers understand what the model learns, generates, and verifies.
Questions for authors
-
What specific methodological advances or new empirical insights does this paper provide beyond PACT, Palermo et al. (2022), and especially Yeh and Ong (2026)?
-
What concrete limitation of existing theorem-proving approaches motivates direct kernel-term modelling, and how do the reported findings address it?
-
Can the authors provide a unified task formulation and pipeline overview specifying model inputs and outputs, serialization and decoding, and verification, while clearly distinguishing training, teacher-forced evaluation, and free-running generation?
Dear reviewers,
First of all, we would like to thank you for spending time reading our paper and writing your reviews. We sincerely apologise for the dense content: most of the writing was polished by LLMs, although we reviewed every sentence. We have revised some sentences following your suggestions, and are withdrawing the submission to revise it further.
Research question (PKC2, phwX). We acknowledge that the research question and the takeaway message are weak. A stronger submission would anchor on an empirical advantage over a baseline, which our current results fail to show.
Presentation (PKC2, phwX). We now define terms such as warm start and reference prefix before using them, and have replaced the contributions table.
Pipeline (phwX). We added a pipeline figure (Figure 2) that separates training, teacher-forced scoring, and free generation with checking.
Extrapolation, learning rate, generation and reference prefixes (2Ey9). We agree with these points, which the paper states as limitations.
Thank you again.