acceptodds
Under review as a conference paper at ICLR 2027

CNLVerifier: A Controlled-Natural-Language Verifier for LLM-Driven Mathematical Reasoning

Abstract

Large language models can generate mathematical proofs in natural language, but such informal proofs are difficult to use as reliable, automatic, and actionable feedback for post-training and inference-time compute. Formal proof assistants provide strong correctness guarantees, but their proof languages are often costly for current models to generate and interact with. We present CNLVerifier (Controlled Natural Language Verifier), a lightweight verifier for proofs written in a controlled, LaTeX-flavored natural language. CNLVerifier provides structured diagnostics that can serve both as a binary acceptance signal and as feedback for self-refinement. We use these signals for verifier-filtered supervised fine-tuning and reinforcement learning. Experiments on miniF2F show that verifier-based training and inference-time compute improve the generation of verifier-accepted CNL proofs.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Math verification, Controlled natural language

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BouKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CNLVerifier, a tool for checking mathematical proofs generated by LLMs in controlled natural language. It provides proof verification and structured feedback and can be used for SFT, RL and proof refinement during the inference.

Critical strengths

1.This paper proposes a controled natral language framework for mathematical proofs. It is easy for human to read and can be checked by machines. It uses structured feedback and external solver, it offers practical engineering value.

2.Experiments on miniF2F show that verifier-based SFT improves proof generation. It improves the test pass@1 from 30.99% to 37.82%, showing the verifier feedback is helpful.

Critical weaknesses

1.Only 15 of 57 tactics can produce kernel derivations or per-instance solver certificates. Only 12.9% of all kernel-backed steps don't depend on external procedures. Also, the CNL-to-FOL translation is not formally verified.

2.The experiments are only conducted on miniF2F. There is no evaluation on verification reliability and generalization. The baseline and ablation studies are not enough. It is difficult to know the contribution of each component.

Questions for authors

1.How to evaluate the use of external procedures and whether the CNL-to-FOL translation is semantically correct?

2.Can the authors provide more ablation studies to show where the performance improvements come from?

Reviewer_stSWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CNLVerifier, a verifier for LLM-generated math proofs written in verifiable and LaTeX-flavored natural language (CNL). Each proof line must follow a small set of sentence templates, and the math expressions are translated into many-sorted first-order logic (FOL). The proof is then verified by a self-implemented kernel. When the verification fails, the verifier returns structured diagnostics that can be used for self-refinement. The environment applies rejection-sampling SFT and GRPO-based RL on Qwen3-4B. The result indicates relatively meaningful performance compared to the base model.

Critical strengths
  1. Interesting middle ground between formal and natural language: The idea of bridging fully verifiable formal languages and natural language (NL) via a controlled subset of NL is compelling. The template-based sentences keep proofs close to how mathematicians write while remaining structured enough for deterministic parsing and verification.
  2. A verifiable environment may benefit training: The verifier provides a well-defined environment for both SFT and RL. It can provide rejection sampling for SFT data collection and RL rewards through verification.
Critical weaknesses
  1. Unclear necessity of CNLVerifier: My primary concern is whether the proposed framework is truly necessary. Several existing works [1, 2] already try to bridge this gap. Beyond a brief statement in Section 2, the paper does not explain why CNLVerifier is better than these alternatives. In particular, it remains unclear which group of users, under what circumstances, would choose CNLVerifier over Lean, Visored [1], or even Isabelle, which are also designed to be readable.
  2. Choice and reliability of the kernel: Another important concern for me is the self-implemented kernel reliability, including the following points:
    1. Firstly, FOL is a less expressive foundation for general mathematics than higher-order logic or dependent types. It requires special care to prevent Russell's paradox, and recursion with non-predefined depth is also tricky to model.
    2. Second, the authors do not clearly explain how the kernel's quality is verified. It only proves some theorems; the quality of the kernel implementation itself is not discussed. The paper also does not report systematic testing or cross-checking of the accepted proofs against an established proof assistant.
Questions for authors
  1. Can the authors clarify the concrete advantage of CNLVerifier compared to other languages in a similar position? For which users and scenarios would CNLVerifier be the preferred choice but not others?
  2. Can the authors explain why they chose FOL as the kernel’s foundation and how the current design would handle mathematics that requires general recursion or stronger forms of induction?
  3. Can the authors provide a detailed analysis and verification of the kernel itself?
Reviewer_7fF4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a restricted, LaTeX-like proof language and custom verifier. It uses verifier acceptance to filter training data, reward reinforcement learning, and guide proof revision.

Critical strengths
  1. The readable syntax and structured diagnostics offer a potentially useful interface for proof generation.
  2. Training improves held-out verifier acceptance, and the authors explicitly document which parts of the system must be trusted.
Critical weaknesses
  1. The reliability of accepted proofs is insufficiently established. Reported results include proofs accepted through tactic success flags without a complete kernel derivation. Other proofs depend on unchecked endorsements. Theorem 4.1 accounts for these dependencies but does not establish their correctness; Table 3 reports step-level dependencies rather than the fraction of independently validated complete proofs. Using this same verifier for training rewards and evaluation therefore leaves unclear whether higher acceptance reflects improved mathematical correctness.
  2. The benefit of introducing a separate language and verifier is not demonstrated. Readable syntax does not by itself establish that a language is easier for LLMs. There is no matched comparison showing better independently verified success or lower cost than an established proof interface. The experiments demonstrate adaptation to CNLVerifier, but do not establish an advantage sufficient to justify the additional trusted implementation.
Questions for authors
  1. What independent validation establishes the correctness of accepted proofs, and do the SFT/RL gains remain after excluding driver-only acceptance and unchecked tactic endorsements?
  2. Under comparable model, training, and inference budgets, what advantage does CNL provide over an established proof interface, and why does that advantage require a separate verifier?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.