- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposed a method for training code agent in the formal verifiable domain (especially on Dafny). From my view, this is a co-design combining with the training reward mechanism and agent harness. While I think the topic is interesting and relatively under-explored, the baseline and evaluation is somehow flaw and needs more fair comparisons or explanations. More details are provided below.
Critical strengths
- Interesting topic. Formal verifiable code agentic RL is not as famous as test-based code agentic RL. I am happy to see more diverse software engineering tasks combining with agentic RL methods.
- The method is straightforward. While I don't think the idea is surprisingly novel, I do think it make sense, and this topic is relatively under-explored.
Critical weaknesses
- Baselines and experiments are flaw. Specifically:
- Comparison with test-time scaling methods. Since authors mentioned that "Training against the verifier instead raises two challenges" in the intro, a fairer comparison should amortize the training compute cost (FLOPS) across each instance in the eval set and then conducting inference-only evaluations for base model using various test-time scaling methods on those problems (for example, major voting.). It would be better to plot the results on a log-scale to determine the inference-only cost required to surpass the performance achieved through training. (Please refer to Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters)
- Lack of related work comparison . While I acknowledge that there is limited RL research in formal verification agent RL, methods such as TAN (mentioned in the related work) could likely be adapted for comparison. The authors should at least explain why they did not compare their work against the agentic RL components of these related methods (after adaptation). If no fair baseline comparison is possible, the reasons should be explained. Otherwise, the methods should be adapted and included in the comparison.
- I also encourage the authors to include a generalization analysis in their baseline comparisons. I think a simple held-out training/evaluation split might suffer from data leakage due to the presence of highly similar problems.
-
The writing is somewhat difficult for humans to read, particularly the abstract. Although the context clarifies that the "verifier" discussed here refers to formal verification, the term "verifier" is frequently used in RL to denote any ground-truth reward rule (e.g., in code tasks, testing is often viewed as a "verifier" rather than "validation" in the traditional program analysis sense). Therefore, I suggest explicitly specifying in the title and abstract that this paper focuses on "formal" verifiers.
-
The authors have not provided an artifact, raising concerns regarding reproducibility. I recommend making the code available.
Questions for authors
- Can you add extra experiments with test-time scaling methods and related work baselines (e.g., TAN), or at least explain why you did not compare with them?
- How are the SWE-Proof success rates in Table 2 calculated? Given one greedy episode per instance, the reported 3.1% and 68.8% cannot arise from 58 annotation-repair instances, nor can 56.3% arise from 109 code-generation instances. Please clarify the denominators and aggregation procedure.
- Can you provide artifacts?
- Can you provide a name for your method?
6 comments
Response to Reviewer rYe5 (2/2)
W4. The writing is somewhat difficult for humans to read, particularly the abstract. Although the context clarifies that the "verifier" discussed here refers to formal verification, the term "verifier" is frequently used in RL to denote any ground-truth reward rule (e.g., in code tasks, testing is often viewed as a "verifier" rather than "validation" in the traditional program analysis sense). Therefore, I suggest explicitly specifying in the title and abstract that this paper focuses on "formal" verifiers.
A4. We thank the reviewer for this helpful suggestion. We have revised the title and abstract to explicitly emphasize that our work focuses on formal program verification, and have improved the abstract's clarity in the revised manuscript.
W5. The authors have not provided an artifact, raising concerns regarding reproducibility. I recommend making the code available.
A5. We thank the reviewer for raising this concern. We have submitted a request to open-source our code and will release it once approved. In the meantime, we have provided detailed implementation information in the appendix to support reproducibility. Specifically, Appendix B describes the agentic environment, available actions, and execution protocol; Appendix C details the reward computation, hidden probes, and integrity checks; Appendix D provides the training hyperparameters, verifier versions, and evaluation protocol; and Appendix E documents the benchmark construction and data splits. Appendix H further provides representative agent trajectories with detailed verifier feedback. We hope these details help facilitate reproducibility of our work.
Q1. Can you add extra experiments with test-time scaling methods and related work baselines (e.g., TAN), or at least explain why you did not compare with them?
A6. Compared with Tan [1], our method additionally trains on annotation repair, uses a dense reward based on discharged proof obligations rather than coarse outcome-based rewards, and incorporates a separately trained frozen proof specialist. We have already evaluated these differences through our experiments, including the binary-reward baseline in Table 3 and the without-specialist ablation in Figure 5. Please refer to A2 for detailed explanations and comparisons.
References:
[1] Tan, M. (2026). Automating Formal Verification with Reinforcement Learning and Recursive Inference. arXiv preprint arXiv:2605.30914.
Q2. How are the SWE-Proof success rates in Table 2 calculated? Given one greedy episode per instance, the reported 3.1% and 68.8% cannot arise from 58 annotation-repair instances, nor can 56.3% arise from 109 code-generation instances. Please clarify the denominators and aggregation procedure.
A7. We thank the reviewer for identifying this discrepancy. The SWE-Proof verification results in Table 2 were evaluated on a randomly sampled subset of 32 instances per task from the full held-out split, rather than the entire held-out set. Each success rate is calculated from one greedy episode per instance, with 1/32 (3.1%) and 22/32 (68.8%) successful instances for annotation repair, and 18/32 (56.3%) and 32/32 (100.0%) for code generation, before and after training, respectively. We have clarified the evaluation subset sizes and aggregation procedure in both the main text and appendix of the revised manuscript.
We have additionally evaluated the initial and RL-trained models on the complete SWE-Proof held-out splits. The results are reported below. The RL-trained model achieves substantial improvements on both tasks, consistent with the findings in Table 2.
| Model | Annotation repair | Code generation |
|---|---|---|
| Init | 3.4 | 45.9 |
| RL | 50.0 | 80.7 |
Q3. Can you provide artifacts?
A8. We have submitted a request to open-source our code and will release the artifacts once approved. In the meantime, we have provided detailed implementation and reproducibility information in the appendix. Please refer to A4 for further details.
Q4. Can you provide a name for your method?
A9. We thank the reviewer for this suggestion. We have named our method VGA-RL (Verifier-Grounded Agentic RL) and introduced this name in the abstract, contributions, and Figure 1 of the revised manuscript.
We hope our response addresses your concerns, and we would appreciate your consideration of updating your assessment in light of our clarifications and additional experiments. We would be happy to address any remaining concerns during the discussion period.
Thank you for your detailed response. I am satisfied with the extra held-out experiment.
The only concern remains here is that the training compute cost (FLOPS) is unknown for comparing your method with test-time scaling baseline. I think it is fine that pass@10 outperforms RL on DafnyBench annotation, but what is the actual cost? To be clear, I am not asking for a test-time scaling baseline matched to the training budget (which is unfair to RL training, as training is paid once, but test-time scaling spreads its cost over every single inference.) I only ask that the two costs be reported honestly, so that a researcher working under a limited compute budget can tell which method is the better choice.
I also recommend the authors to add this comparison with a log-scale curve figure in the appendix.
BTW, I am not sure the meaning of "Pass@10" here. Does it mean the major voting over 10 repeated samples? the best / average effort of 10 samples? or the union of passed instances over these 10 samples?
Response to Reviewer rYe5 (1/2)
We thank Reviewer rYe5 for the constructive review. We appreciate your recognition of the importance of exploring agentic RL for formal verification and the practicality of our proposed approach. We address your concerns as follows.
W1. Comparison with test-time scaling methods. Since authors mentioned that "Training against the verifier instead raises two challenges" in the intro, a fairer comparison should amortize the training compute cost (FLOPS) across each instance in the eval set and then conducting inference-only evaluations for base model using various test-time scaling methods on those problems (for example, major voting.). It would be better to plot the results on a log-scale to determine the inference-only cost required to surpass the performance achieved through training.
A1. We thank the reviewer for this valuable suggestion. To evaluate test-time scaling, we conducted additional experiments using pass@5 and pass@10 on the untrained model. The results are reported below.
Test-time scaling slightly outperforms our RL-trained model on DafnyBench annotation repair, but remains below it on the other three tasks, with particularly large gaps on code generation and SWE-Proof annotation repair. These results demonstrate the benefits of RL training beyond simply increasing inference-time sampling. We also note that test-time scaling is complementary to RL training, as our RL-trained model can likewise benefit from additional inference-time compute.
W2. Lack of related work comparison . While I acknowledge that there is limited RL research in formal verification agent RL, methods such as TAN (mentioned in the related work) could likely be adapted for comparison. The authors should at least explain why they did not compare their work against the agentic RL components of these related methods (after adaptation). If no fair baseline comparison is possible, the reasons should be explained. Otherwise, the methods should be adapted and included in the comparison.
A2. We thank the reviewer for highlighting this important work. Tan [1] differs from our paper in three main aspects: (1) Tan trains only on code generation, whereas we additionally train on annotation repair; (2) Tan uses a coarse outcome-based reward, whereas our dense reward measures progress toward formal verification through the obligation ratio; and (3) Tan does not employ a separately trained proof specialist, whereas we train an annotation-repair specialist and integrate it into the code-generation loop.
Regarding baseline comparisons, we have already evaluated the relevant components of [1] through our baselines and ablations. Specifically, Table 3 compares our method against binary-reward RL, while Figure 5 evaluates the contribution of the frozen proof specialist. Our method outperforms the corresponding baselines in both comparisons. We have also clarified these comparisons in the related work section of the revised manuscript.
References:
[1] Tan, M. (2026). Automating Formal Verification with Reinforcement Learning and Recursive Inference. arXiv preprint arXiv:2605.30914.
W3. I also encourage the authors to include a generalization analysis in their baseline comparisons. I think a simple held-out training/evaluation split might suffer from data leakage due to the presence of highly similar problems.
A3. Thank you for this valuable suggestion. We have conducted additional out-of-distribution evaluations of our baselines on HumanEval-Dafny and HumanEval-Verus, which were entirely excluded from training. The results are reported below.
While some baselines achieve competitive performance on annotation repair, our method substantially outperforms all baselines on code generation, particularly on HumanEval-Verus, where both the programming language and verifier are unseen during training. These findings are consistent with Table 3 and provide further evidence that our improvements generalize beyond the training benchmarks. We have incorporated these results into the revised manuscript.