acceptodds
Withdrawn from ICLR 2027

Language Modelling and Scaling Laws for Lean’s Lambda Calculus

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Typed lambda calculi underlie many proof assistants; e.g., Lean’s kernel checks terms that express proofs and programs, making proof steps explicit. How difficult is this language for an autoregressive model to learn and generate? We compare kernel terms, surface Lean, and informalised English descriptions of the same Mathlib theorems using our developed serialisation method, which follows the dependent type from left to right. The decoder then samples kernel expressions, which can be checked without elaboration. We probe the scaling laws, both from scratch and with a warm start using Qwen3, in each language. For Qwen3-4B warm starts under reference prefixes, the 1% of proof tokens with the highest loss carry 98.6% of term proof loss, against 10.7% for surface Lean and 14.3% for English. Mean term proof loss is 2.475 times surface Lean’s at these warm starts but falls faster with model size and data from scratch, though the grid’s single learning rate slightly inflates the term model-size exponent. If the fits continue with fresh data and model size taken to infinity, term proof loss crosses below surface Lean’s after about 7 times the current pool of distinct proofs. The transferred Qwen3-0.6B achieves a term proof loss lower by a factor of 3.68 than the same network trained from scratch on the same number of tokens. Based on these results, we’re in a position to use pretrained models and increase the dataset size for kernel term language.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, formal mathematics, Lean 4, proof terms, typed lambda calculus, dependent type theory, language modelling, neural theorem proving, tokenisation, fast vocabulary transfer, transfer learning, cross-entropy

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_phwXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies autoregressive modeling of serialized Lean kernel statements and proofs, comparing kernel terms, surface Lean, and model-generated English for the same Mathlib theorems. It develops a deterministic term serialization, trains models from scratch and transfers Qwen3 checkpoints to the new vocabulary, analyzes proof-loss concentration, and fits model/data scaling laws. It also checks sampled proofs with Lean and relates reference-proof likelihood to a lower bound on generation success. The main findings are highly concentrated kernel-term token loss, substantial transfer from pretrained weights, and steeper measured loss-reduction trends for kernel terms, although current surface-Lean generation is stronger.

Critical strengths

1. Useful empirical study. The comparison across three proof representations provides valuable insights.

2. Reliable experiments. Careful verification and baseline controls strengthen the evaluation.

Critical weaknesses
  • Limited novelty. Learning from Lean proof terms is well established in PACT (ICLR 2022), and Palermo et al. (AITP 2022) already trained language models to generate proof terms and filtered outputs using the Lean kernel. More directly, Yeh and Ong (2026) investigated kernel-term predictability and preliminary scaling laws. Therefore, the paper does not sufficiently distinguish these contributions from the closest prior work or establish what substantially new insight they provide beyond extending an existing experimental framework.
  • Unclear motivation and background. The introduction jumps from a vague question about learning a “minimal grammar” to technical details and numerical findings without establishing a clear research problem. It does not adequately explain how kernel-term modelling differs from generating Lean source or tactics, why this distinction matters, or what limitation of existing approaches it addresses. Terms such as “warm start” and “reference prefixes,” together with proof-loss statistics, appear before readers are given sufficient context to interpret them.

Moreover, Table 1 largely repackages the abstract into a text-heavy table. It mixes claimed contributions, numerical results, implementation details, and conditional extrapolations without clearly distinguishing their significance. Overall, this paperlooks like more of an experimental report rather than a scholarly paper.

  • Unclear method formulation. Although the paper defines prediction losses and scaling fits, it lacks a clear, unified formulation of the overall pipeline. The relationships between theorem statements, proof representations, model inputs and outputs, and kernel verification are scattered across the main text and appendices. Training, teacher-forced evaluation, and free-running generation are also not clearly distinguished in a consolidated overview. A precise task definition and an algorithm or pipeline diagram would help readers understand what the model learns, generates, and verifies.
Questions for authors
  1. What specific methodological advances or new empirical insights does this paper provide beyond PACT, Palermo et al. (2022), and especially Yeh and Ong (2026)?

  2. What concrete limitation of existing theorem-proving approaches motivates direct kernel-term modelling, and how do the reported findings address it?

  3. Can the authors provide a unified task formulation and pipeline overview specifying model inputs and outputs, serialization and decoding, and verification, while clearly distinguishing training, teacher-forced evaluation, and free-running generation?

Reviewer_PKC2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper measures token level losses produced by LLMs on proofs written in three different formats, and show that few tokens are responsible for most of the loss. Experiments show that scaling training data reduces prediction error more rapidly for kernel term compared to surface lean and English language proofs.

Critical strengths
  1. Experiments on scaling laws comparing various proof formats are noteworthy.
Critical weaknesses
  1. The writing in this paper is terse and prohibitively high entropy, possibly due to heavy LLM usage. This makes the findings presented difficult to parse, particularly regarding the authors' motivation and intent behind their experiments. Sections 4.1, 4.2, and 4.3 present various metrics with little to no exposition regarding why these numbers are insightful. It is possible that the findings in this paper can be valuable to the community, but they need to be organized and communicated more clearly.
Questions for authors
  1. Some findings from this paper include: a small minority of tokens are responsible for a large portion of token loss in formal proofs, pretrained LLMs learn kernel term format better than random initialization, and pass-at-k performance can be lower bounded by sequence loss for the term model. Can you explain why these results are important, and what research question they answer?
Reviewer_2Ey9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies the learnability and scaling of Lean 4 kernel proof terms, comparing them with surface Lean and LLM-generated English explanations of the same Mathlib theorems.

Contributions:

  • A lossless-up-to-irrelevant-fields, prefix-order serialization of Lean kernel expressions that supports autoregressive generation and direct kernel verification without elaboration
  • Under teacher forcing, the highest-loss 1p of kernel-term tokens account for 98.6p of total proof loss, suggesting that most of the explicit proof structure is highly predictable.
  • Kernel terms exhibit steeper empirical loss reductions with model size and training data than surface Lean or English, although the fitted scaling law extrapolates beyond the observed regime.
Critical strengths

The paper does a systematic comparison of kernel terms, surface Lean, and English on matched mathematical content. The striking concentration of proof loss, with the analysis of premise selection, provides a concrete insight into the structure of formal proof. This is the strongest reason to accept, I think.

The serialization enables generation of complete kernel-checkable expressions, rather than relying on surface-level elaboration. The controlled Qwen3 comparison demonstrates substantial benefits from pretrained weights. Together, these contributions establish a useful experimental foundation for further work on direct proof-term generation.

Critical weaknesses

The fitted model provides a substantially worse fit for kernel terms than for the other representations, and kernel-term loss can steepen with additional data, contrary to the fitted functional form. The claimed crossover with surface Lean requires extrapolation to N -> inf and approximately seven times the existing pool of distinct proofs. So the extrapolated crossover is not reliable evidence that kernel terms will eventually outperform surface Lean. Although the observed steeper improvements are interesting, the extrapolation should not be central to the conclusions. I believe the main scaling-law extrapolation is thus insufficiently supported.

It seems kernel terms remain considerably less successful than surface Lean in verified generation. More importantly, every miniF2F problem solved by the kernel-term model is also solvable using the six simple templates. Thus, the experiments establish efficient modeling of explicit proof structures but offer limited evidence of improved mathematical reasoning or nontrivial proof discovery. The concentration results are also measured under reference prefixes, which need not characterize errors during autoregressive generation.

Questions for authors

Clarifications / additional analysis that resolve some of the main concerns above would be helpful. For example, some questions worth answering might be: Given the substantial fit error and learning-rate sensitivity, how robust is the predicted kernel-term/surface-Lean crossover under alternative scaling-law functional forms and optimization-controlled comparisons? In particular, is there evidence supporting a crossover at realistically finite model sizes? Among the Mathlib theorems solved by kernel-term generation but not by the six templates, how many require genuinely different mathematical arguments or previously unseen premise combinations?

Comments on the paper

1 comment

Dear reviewers,

First of all, we would like to thank you for spending time reading our paper and writing your reviews. We sincerely apologise for the dense content: most of the writing was polished by LLMs, although we reviewed every sentence. We have revised some sentences following your suggestions, and are withdrawing the submission to revise it further.

Research question (PKC2, phwX). We acknowledge that the research question and the takeaway message are weak. A stronger submission would anchor on an empirical advantage over a baseline, which our current results fail to show.

Presentation (PKC2, phwX). We now define terms such as warm start and reference prefix before using them, and have replaced the contributions table.

Pipeline (phwX). We added a pipeline figure (Figure 2) that separates training, teacher-forced scoring, and free generation with checking.

Extrapolation, learning rate, generation and reference prefixes (2Ey9). We agree with these points, which the paper states as limitations.

Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.