- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CNLVerifier, a tool for checking mathematical proofs generated by LLMs in controlled natural language. It provides proof verification and structured feedback and can be used for SFT, RL and proof refinement during the inference.
Critical strengths
1.This paper proposes a controled natral language framework for mathematical proofs. It is easy for human to read and can be checked by machines. It uses structured feedback and external solver, it offers practical engineering value.
2.Experiments on miniF2F show that verifier-based SFT improves proof generation. It improves the test pass@1 from 30.99% to 37.82%, showing the verifier feedback is helpful.
Critical weaknesses
1.Only 15 of 57 tactics can produce kernel derivations or per-instance solver certificates. Only 12.9% of all kernel-backed steps don't depend on external procedures. Also, the CNL-to-FOL translation is not formally verified.
2.The experiments are only conducted on miniF2F. There is no evaluation on verification reliability and generalization. The baseline and ablation studies are not enough. It is difficult to know the contribution of each component.
Questions for authors
1.How to evaluate the use of external procedures and whether the CNL-to-FOL translation is semantically correct?
2.Can the authors provide more ablation studies to show where the performance improvements come from?