- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the joint generation of executable code, formal specifications, and proofs in Verus. It introduces SpecBench, a 97-task specification-restoration benchmark, and VerusLM-4B-Codex, trained through multicomponent supervised fine-tuning and multistage verifier-guided reinforcement learning. Experiments cover proof generation, specification restoration, and end-to-end generation across VeruSage-Bench, SpecBench, VTB-XS and VerifyThisBench. The trained model achieves higher verifier pass rates, but semantic validation reveals substantial remaining gaps: it obtains 0% specification-equivalence success on SpecBench and 0% coherence on VerifyThisBench despite a 75.61% verifier pass rate there. These findings highlight persistent reward hacking and the importance of evaluating specification quality alongside proof acceptance.
Critical strengths
-
Solid experiments. The evaluation covers 4 different benchmarks and compares performance against both general-purpose and specialized models.
-
Straightforward method. The approach is fairly direct and easy to follow.
Critical weaknesses
-
Bad writing. The "AI USE STATEMENT" section is missing, which violates the ICLR submission format. Broken references such as "Section ??" and "Table ??" appear in both the main text and the appendix. In Section 4.3, there is a missing line break before "Multi-turn environment." Overall, the paper feels rushed.
-
After RL, the Plain-Harness metric on VeruSage-Bench and the Coherent metric on VerifyThisBench decline. At the end of Section 5.2.1, the authors also note that "reward-hacking pattern persists whenever the verification signal dominates supervision." This seems to suggest that the task may not be well suited to reinforcement learning for model training, since judging the correctness of model-generated spec is highly difficult.
Questions for authors
Missing necessary ablation or comparison experiments. How do the data mixing ratio and Gated reward affect RL performance? Also, would training methods that are effective on Lean [1] or Dafny [2] outperform the proposed method when applied to Verus?
[1] Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verification. arxiv.org/abs/2603.19329
[2] Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny. https://arxiv.org/abs/2507.16331