- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes AuXSpec, a framework that enables test-based feedback for training LLMs to generate formal program specifications in Lean 4. The key idea is to pair declarative postconditions with executable counterparts, allowing test cases to provide automated and reliable reward signals for specification generation. The authors construct a large-scale dataset and train models using supervised fine-tuning and reinforcement learning, achieving substantial improvements on both in-domain and out-of-domain benchmarks.
Critical strengths
-
Important problem. Automated ways to check the correctness of LLM-generated specifications are crucial to LLM-based automated program verification.
-
Bringing established PL/FV techniques to the AI community. Executable specifications are a well-established technique in the Programming Languages (PL) and Formal Verification (FV) communities. This is the right way to evaluate LLM-generated specifications and improve LLM-based specification generation. I am fully convinced because execution of specifications is standard in the PL/FV community. I appreciate the paper's merit in bringing this approach to the AI community and integrating it with LLM-based specification generation and training.
-
Clear and careful presentation. The paper is clearly and carefully written. I particularly appreciate its attention to potential pitfalls and detailed methodological explanations. It makes me feel I'm reading a rigorously written PL article and I enjoy it.
Critical weaknesses
- In the end, the paper integrates established techniques from the PL/FV and AI communities, rather than introducing a fundamental breakthrough. This is not to say that the work lacks novelty. Given the substantial gap between these two research communities, bringing well-established PL/FV techniques into the AI community is itself a valuable contribution.
- The paper should mention in its Related Work section the long-standing research history of executable specifications in the PL/FV communities.
- Figure 1 is too small.
Questions for authors
I have no questions.
3 comments
Dear Reviewer mWZx,
Thank you again for the encouraging review and for pointing us to the history of executable specifications, which we learnt a lot through reading. The discussion period ends on November 18 (AoE), and we would like to know whether the revision addresses your comments. The changes are marked in blue in the revised PDF.
W2: Research history of executable specifications
- We rewrite the related work paragraph on executable specifications to place AuXSpec within this history (Section 6, page 10). You can read the full paragraph with all references in Section 6.
- The introduction now points to this connection where the two postconditions are first described (page 2).
W3: Size of Figure 1
- Following your suggestion, we redraw Figure 1 as a compact diagram of the pipeline (Section 3, page 4).
- The submitted figure moves to Appendix G as Figure 12 at full text width, next to the description of each stage. A page view shows that the figure stays legible at this size.
W1 and further experiments: the value of executable feedback
Since your review, we have also run more detailed ablations of the RL recipe, which show that test execution provides a necessary training signal (Appendix F.2).
- Without test execution, both generation tasks improve more slowly, and CodeGen falls furthest behind, since no other stage of the reward runs the implementation.
A matched run that skips test execution and keeps all other parts of the recipe falls behind AuXSpec-9B-RL by 5.0 and 10.6 points in SpecGen and by 19.5 and 30.6 points in CodeGen. Its reward also accepts implementations that the test cases would reject (49.7% vs. 19.7% of CodeGen rollouts accepted in steps 41 to 50).
- With the execution-based reward, SFT on a small dataset followed by RL outperforms SFT alone on far larger data.
We validate this by training the same recipe on Qwen3-4B, the backbone of VeriSpecGen-4B, with 19.5K SFT samples. After SFT our model scores 6.9 SpecGen pass@1 on Verina, and RL improves this score to 54.2. This 4B model outperforms the 30B-A3B models of VeriSpecGen (47.6 pass@1 at best), which train with SFT only on 122K to 344K distilled samples. This comparison supports the effectiveness of executable postconditions for specification generation (Appendix F.1).
We would be happy to discuss anything further before the deadline. If the revision addresses your concerns, we would be grateful if you could consider updating your rating.
Best regards,
The authors
I am satisfied with the authors’ response and will increase my score. In fact, my original inclination was a 4, but I wanted to first consider the opinions of the other reviewers. Given the rebuttal and the overall discussion, I no longer find a reason to maintain a weak accept.
Response
Thank you for the positive assessment and for pointing us to the history of executable specifications in the PL and formal verification communities. In the revised PDF, the following changes are marked in blue:
W1: Bringing established PL/FV techniques to LLM specification generation.
We agree with this view of the contribution. AuXSpec applies executable specifications to training models that generate Lean specifications from natural-language problems. The models generate both the declarative and the executable postcondition. Running the executable postcondition on test cases filters the training data and forms part of the reinforcement-learning reward.
W2: Research history of executable specifications.
We have added a paragraph on this history to the related work. A short summary follows, and the full paragraph with all references is in Section 6.
The executability of specifications has long been debated in formal methods. Executable specifications can be validated against intended behavior by execution (Kemmerer, 1985; Fuchs, 1992), whereas declarative specifications are more expressive and independent of any algorithm (Hayes & Jones, 1989).
AuXSpec combines both sides of this trade-off. Models generate the executable variants, a step that previously required manual curation. In our curated corpus, 35.5% of the executable postconditions come from decidability instances in Lean and agree with the declarative postcondition by construction. For the other 64.5% models generate a separate executable postcondition. The postcondition equivalence theorem then certifies the agreement, as the agreement proof does in Boolean reflection. This theorem is provable as generated for 99.0% of the 3,037 synthesized problems. The revised introduction now points to this connection where the two postconditions are first described.
W3: Figure 1 is too small.
We redrew Figure 1 as a compact diagram of the stages, the frozen inputs and the accepted components. The submitted figure pairs each stage with the input of a running example and the Lean code that the stage produces. It now appears at the start of Appendix G as Figure 12 at the full text width, next to the detailed description of each stage. At this size the figure remains legible (page view).
Thank you again for the thoughtful review. We learned a lot from reading the relevant literature, which made the connection to PL/FV much clearer. Hoping these refinement would address your concerns, and we are glad to discuss them further!