acceptodds
Under review as a conference paper at ICLR 2027

Verifier-Grounded Agentic RL for Provably Correct Code

Abstract

Reinforcement learning for code usually rewards passing tests, which a model can do while remaining wrong on the inputs the tests do not cover. A program verifier instead decides whether a program satisfies a formal specification for all inputs. Language models are almost always paired with verifiers at inference time only, and turning a verifier into a reward raises two difficulties. It reports only acceptance or rejection, which gives no gradient when every attempt fails, and a policy can cheat it, most simply by telling it to assume what the proof was supposed to establish. We build an agentic environment and reward that address both. The reward reads the verifier's diagnostics rather than its verdict, scoring a candidate by the weighted fraction of proof obligations it discharges, extended with hidden probes from a reference proof's annotations. An integrity audit outranks the verifier, rejecting any candidate that alters the specification or assumes what it was meant to prove. The model works as an agent against a live verifier over many turns, with a separately trained proof specialist as a frozen collaborator. Across Qwen3 models from 1.7B to 14B and a 35B Qwen3.6 mixture-of-experts model, held-out verification on code generation rises from 66.1% to 93.5% on DafnyBench and from 56.3% to 100% on SWE-Proof. This reward leads every other training signal we compare, including a binary pass-or-fail reward trained under otherwise identical settings, with the widest margin on code generation. When the agent must also deliver the repository patch, graded by each project's withheld tests, the fraction of held-out issues it resolves rises from 59.6% to 72.5%. Evaluated unchanged on three verification suites withheld from training, the checkpoints improve on all of them, including one whose language and verifier are absent from training.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Formal Verification, Agentic Reinforcement Learning, Coding Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rYe5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposed a method for training code agent in the formal verifiable domain (especially on Dafny). From my view, this is a co-design combining with the training reward mechanism and agent harness. While I think the topic is interesting and relatively under-explored, the baseline and evaluation is somehow flaw and needs more fair comparisons or explanations. More details are provided below.

Critical strengths
  • Interesting topic. Formal verifiable code agentic RL is not as famous as test-based code agentic RL. I am happy to see more diverse software engineering tasks combining with agentic RL methods.
  • The method is straightforward. While I don't think the idea is surprisingly novel, I do think it make sense, and this topic is relatively under-explored.
Critical weaknesses
  1. Baselines and experiments are flaw. Specifically:
  • Comparison with test-time scaling methods. Since authors mentioned that "Training against the verifier instead raises two challenges" in the intro, a fairer comparison should amortize the training compute cost (FLOPS) across each instance in the eval set and then conducting inference-only evaluations for base model using various test-time scaling methods on those problems (for example, major voting.). It would be better to plot the results on a log-scale to determine the inference-only cost required to surpass the performance achieved through training. (Please refer to Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters)
  • Lack of related work comparison . While I acknowledge that there is limited RL research in formal verification agent RL, methods such as TAN (mentioned in the related work) could likely be adapted for comparison. The authors should at least explain why they did not compare their work against the agentic RL components of these related methods (after adaptation). If no fair baseline comparison is possible, the reasons should be explained. Otherwise, the methods should be adapted and included in the comparison.
  • I also encourage the authors to include a generalization analysis in their baseline comparisons. I think a simple held-out training/evaluation split might suffer from data leakage due to the presence of highly similar problems.
  1. The writing is somewhat difficult for humans to read, particularly the abstract. Although the context clarifies that the "verifier" discussed here refers to formal verification, the term "verifier" is frequently used in RL to denote any ground-truth reward rule (e.g., in code tasks, testing is often viewed as a "verifier" rather than "validation" in the traditional program analysis sense). Therefore, I suggest explicitly specifying in the title and abstract that this paper focuses on "formal" verifiers.

  2. The authors have not provided an artifact, raising concerns regarding reproducibility. I recommend making the code available.

Questions for authors
  • Can you add extra experiments with test-time scaling methods and related work baselines (e.g., TAN), or at least explain why you did not compare with them?
  • How are the SWE-Proof success rates in Table 2 calculated? Given one greedy episode per instance, the reported 3.1% and 68.8% cannot arise from 58 annotation-repair instances, nor can 56.3% arise from 109 code-generation instances. Please clarify the denominators and aggregation procedure.
  • Can you provide artifacts?
  • Can you provide a name for your method?
6 comments
AuthorsOpenReview ↗

Response to Reviewer rYe5 (1/2)

We thank Reviewer rYe5 for the constructive review. We appreciate your recognition of the importance of exploring agentic RL for formal verification and the practicality of our proposed approach. We address your concerns as follows.


W1. Comparison with test-time scaling methods. Since authors mentioned that "Training against the verifier instead raises two challenges" in the intro, a fairer comparison should amortize the training compute cost (FLOPS) across each instance in the eval set and then conducting inference-only evaluations for base model using various test-time scaling methods on those problems (for example, major voting.). It would be better to plot the results on a log-scale to determine the inference-only cost required to surpass the performance achieved through training.

A1. We thank the reviewer for this valuable suggestion. To evaluate test-time scaling, we conducted additional experiments using pass@5 and pass@10 on the untrained model. The results are reported below.

MethodDafnyBench annotationDafnyBench codegenSWE-Proof annotationSWE-Proof codegen
Init67.466.13.156.3
pass@578.378.212.590.6
pass@1080.679.821.993.8
RL76.793.568.8100.0

Test-time scaling slightly outperforms our RL-trained model on DafnyBench annotation repair, but remains below it on the other three tasks, with particularly large gaps on code generation and SWE-Proof annotation repair. These results demonstrate the benefits of RL training beyond simply increasing inference-time sampling. We also note that test-time scaling is complementary to RL training, as our RL-trained model can likewise benefit from additional inference-time compute.


W2. Lack of related work comparison . While I acknowledge that there is limited RL research in formal verification agent RL, methods such as TAN (mentioned in the related work) could likely be adapted for comparison. The authors should at least explain why they did not compare their work against the agentic RL components of these related methods (after adaptation). If no fair baseline comparison is possible, the reasons should be explained. Otherwise, the methods should be adapted and included in the comparison.

A2. We thank the reviewer for highlighting this important work. Tan [1] differs from our paper in three main aspects: (1) Tan trains only on code generation, whereas we additionally train on annotation repair; (2) Tan uses a coarse outcome-based reward, whereas our dense reward measures progress toward formal verification through the obligation ratio; and (3) Tan does not employ a separately trained proof specialist, whereas we train an annotation-repair specialist and integrate it into the code-generation loop.

Regarding baseline comparisons, we have already evaluated the relevant components of [1] through our baselines and ablations. Specifically, Table 3 compares our method against binary-reward RL, while Figure 5 evaluates the contribution of the frozen proof specialist. Our method outperforms the corresponding baselines in both comparisons. We have also clarified these comparisons in the related work section of the revised manuscript.

References:

[1] Tan, M. (2026). Automating Formal Verification with Reinforcement Learning and Recursive Inference. arXiv preprint arXiv:2605.30914.


W3. I also encourage the authors to include a generalization analysis in their baseline comparisons. I think a simple held-out training/evaluation split might suffer from data leakage due to the presence of highly similar problems.

A3. Thank you for this valuable suggestion. We have conducted additional out-of-distribution evaluations of our baselines on HumanEval-Dafny and HumanEval-Verus, which were entirely excluded from training. The results are reported below.

MethodHumanEval-Dafny annotationHumanEval-Dafny codegenHumanEval-Verus annotationHumanEval-Verus codegen
Untrained policy54.567.99.517.2
SFT on reference proofs30.932.88.37.5
SFT on agentic traces52.762.614.38.6
Binary outcome reward72.764.114.317.2
Verifier-grounded reward70.092.413.136.6

While some baselines achieve competitive performance on annotation repair, our method substantially outperforms all baselines on code generation, particularly on HumanEval-Verus, where both the programming language and verifier are unseen during training. These findings are consistent with Table 3 and provide further evidence that our improvements generalize beyond the training benchmarks. We have incorporated these results into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer rYe5 (2/2)

W4. The writing is somewhat difficult for humans to read, particularly the abstract. Although the context clarifies that the "verifier" discussed here refers to formal verification, the term "verifier" is frequently used in RL to denote any ground-truth reward rule (e.g., in code tasks, testing is often viewed as a "verifier" rather than "validation" in the traditional program analysis sense). Therefore, I suggest explicitly specifying in the title and abstract that this paper focuses on "formal" verifiers.

A4. We thank the reviewer for this helpful suggestion. We have revised the title and abstract to explicitly emphasize that our work focuses on formal program verification, and have improved the abstract's clarity in the revised manuscript.


W5. The authors have not provided an artifact, raising concerns regarding reproducibility. I recommend making the code available.

A5. We thank the reviewer for raising this concern. We have submitted a request to open-source our code and will release it once approved. In the meantime, we have provided detailed implementation information in the appendix to support reproducibility. Specifically, Appendix B describes the agentic environment, available actions, and execution protocol; Appendix C details the reward computation, hidden probes, and integrity checks; Appendix D provides the training hyperparameters, verifier versions, and evaluation protocol; and Appendix E documents the benchmark construction and data splits. Appendix H further provides representative agent trajectories with detailed verifier feedback. We hope these details help facilitate reproducibility of our work.


Q1. Can you add extra experiments with test-time scaling methods and related work baselines (e.g., TAN), or at least explain why you did not compare with them?

A6. Compared with Tan [1], our method additionally trains on annotation repair, uses a dense reward based on discharged proof obligations rather than coarse outcome-based rewards, and incorporates a separately trained frozen proof specialist. We have already evaluated these differences through our experiments, including the binary-reward baseline in Table 3 and the without-specialist ablation in Figure 5. Please refer to A2 for detailed explanations and comparisons.

References:

[1] Tan, M. (2026). Automating Formal Verification with Reinforcement Learning and Recursive Inference. arXiv preprint arXiv:2605.30914.


Q2. How are the SWE-Proof success rates in Table 2 calculated? Given one greedy episode per instance, the reported 3.1% and 68.8% cannot arise from 58 annotation-repair instances, nor can 56.3% arise from 109 code-generation instances. Please clarify the denominators and aggregation procedure.

A7. We thank the reviewer for identifying this discrepancy. The SWE-Proof verification results in Table 2 were evaluated on a randomly sampled subset of 32 instances per task from the full held-out split, rather than the entire held-out set. Each success rate is calculated from one greedy episode per instance, with 1/32 (3.1%) and 22/32 (68.8%) successful instances for annotation repair, and 18/32 (56.3%) and 32/32 (100.0%) for code generation, before and after training, respectively. We have clarified the evaluation subset sizes and aggregation procedure in both the main text and appendix of the revised manuscript.

We have additionally evaluated the initial and RL-trained models on the complete SWE-Proof held-out splits. The results are reported below. The RL-trained model achieves substantial improvements on both tasks, consistent with the findings in Table 2.

ModelAnnotation repairCode generation
Init3.445.9
RL50.080.7

Q3. Can you provide artifacts?

A8. We have submitted a request to open-source our code and will release the artifacts once approved. In the meantime, we have provided detailed implementation and reproducibility information in the appendix. Please refer to A4 for further details.


Q4. Can you provide a name for your method?

A9. We thank the reviewer for this suggestion. We have named our method VGA-RL (Verifier-Grounded Agentic RL) and introduced this name in the abstract, contributions, and Figure 1 of the revised manuscript.


We hope our response addresses your concerns, and we would appreciate your consideration of updating your assessment in light of our clarifications and additional experiments. We would be happy to address any remaining concerns during the discussion period.

Thank you for your detailed response. I am satisfied with the extra held-out experiment.

The only concern remains here is that the training compute cost (FLOPS) is unknown for comparing your method with test-time scaling baseline. I think it is fine that pass@10 outperforms RL on DafnyBench annotation, but what is the actual cost? To be clear, I am not asking for a test-time scaling baseline matched to the training budget (which is unfair to RL training, as training is paid once, but test-time scaling spreads its cost over every single inference.) I only ask that the two costs be reported honestly, so that a researcher working under a limited compute budget can tell which method is the better choice.

I also recommend the authors to add this comparison with a log-scale curve figure in the appendix.


BTW, I am not sure the meaning of "Pass@10" here. Does it mean the major voting over 10 repeated samples? the best / average effort of 10 samples? or the union of passed instances over these 10 samples?

Reviewer_SxDtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of training language models to generate formally verified code using reinforcement learning (RL). It addresses two difficulties of turning a verifier into a reward -- first, the existing binary verification rewards are sparse as most attempts fail, and second, that the policy can cheat it by weakening specs or adding assumptions of what the proof was supposed to establish. The paper proposes a dense reward based on the fraction of satisfied proof obligations, coupled with integrity audit that rejects the candidates alter the specs or introduce unjustified assumptions. They also introduce a frozen proof specialist that helps the model generate proof annotations. Experiments on DafnyBench and SWE-Proof show significant improvement in verification rates across different model sizes, outperforming binary RL reward and supervised fine-tuning.

Critical strengths
  • Important problem and practical solution. The paper tackles important problem of post-training language models to generate formally verified code, and using verifier diagnostic to provide dense RL rewards is a promising approach. The paper also adds integrity checks that address reward hacking.
  • Strong experimental results. The paper has extensive model coverage, from 1.7B to 35B on DafnyBench, with additional results on SWE-Proof, and the results show significant improvements. The proposed reward also outperforms binary-reward RL. There are additional experiments for the proof specialist and evals on 3 unseen benchmarks that show generalization.
Critical weaknesses
  • Methodology. It is unclear whether the obligation ratio is the best proxy for task progress. The paper fixes the number of proof obligations before training, but different implementations may generate different verification conditions. It is also unclear how reliable the reward is across different programs and benchmarks and how the reward hacking can still be present. E.g., in Appendix H.1, turn 3, the model gets an obligation ratio of 1.0 however it fails the verification (lines 2201-2208). Also, the motivation for the exact reward formula could be justified better -- it is unclear how sensitive the results are to the parameter choices and the limitations of the reward.

  • Evaluation. The evaluation is extensive, but more ablations would be helpful, specifically, the paper could evaluate the individual contributions of the category weights, the hidden probes and integrity penalties to see how each part influences the result. Some additional baseline rewards other than binary would be also useful. The paper also selects the best checkpoint using the held-out evaluation set every 5 training steps, which could add selection bias, especially with small sets of ~100 tasks (both on DafnyBench and SWE-Proof). Lastly, it would be great to see more detailed empirical analysis of failure cases to understand the limitations of the method.

  • Minor, noncritical: some of the parts in the presentation could be improved. For example, the contribuition paragraph would be better if highlighted 3 main contributions (instead of 6) and separately discuss them, e.g. in bullet-points, to ensure clarity. It would be useful to simplify the figures and reduce the information density, e.g., of the figure 1, highlighting the most essential parts. The general writing across the manuscript could also be improved for clarity.

Questions for authors
  • Q1\textbf{Q1}. Have you observed reward hacking not covered by the integrity checks? E.g., could the issue from Appendix H.1, turn 3 be used to increase the obligation ratio without completing the proof? Is it possible to have a structured analysis of failures?
  • Q2\textbf{Q2}. Why do you use the held-out eval set for checkpoint selection instead of a separate validation set?
  • Q3\textbf{Q3}. How reliable is the obligation ratio as a proxy of task progress, given different implementations have different verification conditions? What could be the alternatives here?
  • Q4\textbf{Q4}. How sensitive are the results to the chosen reward coefficients? Could you provide additional ablations?
5 comments
AuthorsOpenReview ↗

Response to Reviewer SxDt (1/3)

We thank Reviewer SxDt for the constructive review. We appreciate your recognition of the importance of our problem, the value of our dense reward and integrity checks, and the strong experimental results across model sizes, tasks, and benchmarks. We address your concerns as follows.


W1. Methodology. It is unclear whether the obligation ratio is the best proxy for task progress. The paper fixes the number of proof obligations before training, but different implementations may generate different verification conditions. It is also unclear how reliable the reward is across different programs and benchmarks and how the reward hacking can still be present. E.g., in Appendix H.1, turn 3, the model gets an obligation ratio of 1.0 however it fails the verification (lines 2201-2208). Also, the motivation for the exact reward formula could be justified better -- it is unclear how sensitive the results are to the parameter choices and the limitations of the reward.

A1. Our obligation ratio provides a denser and more semantically meaningful reward signal than binary verification rewards by measuring how many proof obligations have been discharged toward full verification. While it may not be optimal, it provides useful feedback even when a candidate fails verification. Importantly, the obligations used for reward computation are fixed before training and do not depend on the agent's implementation. For annotation repair, they are derived from the specification and ground-truth implementation; for code generation, they are derived from the specification alone. Different implementations therefore cannot change the obligations being scored.

The reviewer correctly identifies an edge case in Appendix H.1, Turn 3, where the agent introduces an assertion that Dafny subsequently treats as an assumption, but its failure is not captured by our diagnostic parser. This results in an obligation ratio of 1.0 despite failed verification. However, the candidate does not receive the full-verification bonus. We have fixed this edge case in our codebase. Such cases are rare and do not affect our main conclusions, since our primary evaluation metric is the full verification rate, which requires successful verification and is unaffected by this issue.

Regarding the reward coefficients, they were chosen heuristically to prioritize complete, sound verification over partial progress or unsound solutions, rather than cherry-picked for performance. We acknowledge that we have not exhaustively tested their sensitivity, as comprehensive ablations would require substantial additional RL training compute. Nevertheless, the chosen coefficients consistently yield strong performance across model sizes, tasks, and benchmarks, as demonstrated by our experiments.

AuthorsOpenReview ↗

Response to Reviewer SxDt (2/3)

W2. Evaluation. The evaluation is extensive, but more ablations would be helpful, specifically, the paper could evaluate the individual contributions of the category weights, the hidden probes and integrity penalties to see how each part influences the result. Some additional baseline rewards other than binary would be also useful. The paper also selects the best checkpoint using the held-out evaluation set every 5 training steps, which could add selection bias, especially with small sets of ~100 tasks (both on DafnyBench and SWE-Proof). Lastly, it would be great to see more detailed empirical analysis of failure cases to understand the limitations of the method.

A2. We agree that additional ablations on the category weights, hidden probes, and integrity penalties would be valuable. However, each ablation requires substantial RL training compute, making it infeasible to complete these experiments within the rebuttal period. Regarding alternative reward baselines, to our knowledge, existing work primarily uses outcome-based verification rewards [1]. Our binary-reward baseline provides a controlled comparison that isolates the benefit of our dense reward under otherwise identical training settings.

We acknowledge the concern about checkpoint selection bias. To address this, we report the full held-out evaluation curves in Figure 4 and evaluate the trained models on three additional out-of-distribution benchmarks that were never used during training or checkpoint selection. Table 2 shows substantial improvements across these unseen benchmarks, while Figure 4 demonstrates consistent upward trends in held-out verification rates throughout training. These results indicate that the improvements are not artifacts of selecting a favorable checkpoint.

Finally, we have provided both quantitative analyses of failure-related behavior and detailed case studies of unsuccessful trajectories. Appendix F.3 shows that training substantially improves the agent's ability to recover from failed attempts, while specification-altering submissions become increasingly rare. Appendix H.9 further examines a major remaining failure mode, where the agent repeatedly attempts to resolve a difficult proof obligation but ultimately exhausts its turn budget. We hope these analyses address the reviewer's concern about the limitations of our method.

References:

[1] Tan, M. (2026). Automating Formal Verification with Reinforcement Learning and Recursive Inference. arXiv preprint arXiv:2605.30914.


W3. Minor, noncritical: some of the parts in the presentation could be improved. For example, the contribution paragraph would be better if highlighted 3 main contributions (instead of 6) and separately discuss them, e.g. in bullet-points, to ensure clarity. It would be useful to simplify the figures and reduce the information density, e.g., of the figure 1, highlighting the most essential parts. The general writing across the manuscript could also be improved for clarity.

A3. We thank the reviewer for these suggestions. In the revised manuscript, we have reorganized the contribution paragraph into three bullet points, simplified Figure 1 to highlight the key components of our method, and improved the clarity of the writing in several sections.


Q1. Have you observed reward hacking not covered by the integrity checks? E.g., could the issue from Appendix H.1, turn 3 be used to increase the obligation ratio without completing the proof? Is it possible to have a structured analysis of failures?

A4. As discussed in A1, Turn 3 of Appendix H.1 illustrates an edge case that does not affect our primary evaluation metric, the full verification rate. We have also fixed this issue in our codebase. Regarding failure analysis, Appendix F.3 provides quantitative analyses of failure-related behavior, while Appendix H presents detailed case studies of unsuccessful trajectories and their underlying causes. We hope these analyses address the reviewer's concerns.


Q2. Why do you use the held-out eval set for checkpoint selection instead of a separate validation set?

A5. We did not create a separate validation split to preserve as many instances as possible for training, since we also evaluate on three out-of-distribution benchmarks that are entirely excluded from training and checkpoint selection. Furthermore, Figure 4 reports the held-out performance throughout training, showing consistent upward trends rather than improvements limited to the selected checkpoints.

AuthorsOpenReview ↗

Response to Reviewer SxDt (3/3)

Q3. How reliable is the obligation ratio as a proxy of task progress, given different implementations have different verification conditions? What could be the alternatives here?

A6. As explained in A1, the obligations used to compute the reward are fixed independently of the agent's implementation, so the obligation ratio provides a consistent measure of progress across different implementations. More fine-grained rewards could potentially be developed instead of assigning fixed weights to different obligation categories, but investigating such alternatives is beyond the scope of this paper.


Q4. How sensitive are the results to the chosen reward coefficients? Could you provide additional ablations?

A7. We agree that additional ablations on the reward coefficients would be valuable. However, each ablation requires substantial RL training compute, making it infeasible to complete these experiments within the rebuttal period. The coefficients were chosen heuristically to encourage sound, complete verification rather than cherry-picked for performance. As demonstrated by our experiments, these choices have yielded strong and consistent improvements across different model sizes, tasks, and benchmarks.


We hope our response addresses your concerns. If you have further concerns, we will be happy to address them during the discussion period.

Reviewer_ATTfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a verifier-grounded reinforcement learning framework to teach large language models to generate formally verified code. To overcome the sparse, binary feedback of traditional verifiers, the authors convert the verifier's obligation diagnostics into a dense, graded reward signal, allowing the model to receive partial credit for incomplete proofs. Operating in a multi-turn agentic loop alongside a frozen proof specialist, the model iteratively refines its code. Crucially, to prevent reward hacking (e.g., weakening specifications or inserting unchecked assumptions), the environment includes a strict integrity audit that outranks the verifier's verdict. Evaluated on Qwen3 models, the 35B model achieved state-of-the-art verification rates on DafnyBench and SWE-Proof, demonstrating strong capabilities in both code generation and proof annotation.

Critical strengths

Novel Dense Reward Formulation: Elegantly solves the sparse-reward bottleneck in formal verification by converting failed proof obligations into a quantitative metric, enabling RL to progress even when early attempts fail entirely.

Robust Mitigation of Reward Hacking: The integrity audit effectively patches a critical vulnerability in verification-RL by strictly enforcing original specifications and rejecting unsound shortcuts, ensuring that the model actually solves the problem rather than cheating the verifier.

Out-of-Distribution Generalization: The model demonstrates impressive zero-shot transfer to entirely unseen benchmarks and proof languages (such as Rust/Verus), indicating that it learns general verifier-interaction strategies rather than simply memorizing benchmark idioms.

Critical weaknesses

Artificially Weak SFT Baseline: The Supervised Fine-Tuning (SFT) baseline is trained solely on final, polished reference proofs rather than multi-turn problem-solving trajectories. This makes it an uneven comparison against an RL agent that learns from multi-turn interaction and error recovery.

Missing Expert Iteration Comparison: The evaluation lacks a comparison against a highly scalable off-policy method like Expert Iteration or Rejection Sampling. Given the high compute costs of on-policy GRPO, filtering large-scale asynchronous rollouts using the proposed dense reward for iterative SFT might be a highly competitive and faster alternative.

Questions for authors

SFT Data Construction: The current SFT baseline fails to outperform the initial policy because it is trained on final artifacts rather than interactive recovery traces. Could you discuss the feasibility of constructing a "cold-start" dataset of multi-round verifier-guided traces—perhaps by using your dense reward to filter rejection sampling—to train a more representative and robust SFT baseline?

Off-Policy vs. On-Policy Efficiency: How do you anticipate the proposed GRPO approach would compare against an Expert Iteration pipeline? Specifically, utilizing large-scale asynchronous rollouts to collect successful or high-partial-credit trajectories, followed by lightweight, iterative offline SFT. Could this off-policy approach be a more compute-efficient way to utilize the dense reward signal?

Nature of the Transfer Learning: Regarding the impressive transfer to HumanEval-Verus, is it possible that the observed performance gain primarily stems from the model adapting to the rigid "agentic format" (e.g., learning to emit valid tool tags, handle strictly structured outputs, and avoid unparseable syntax) rather than acquiring deep, language-agnostic verification skills?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ATTf

We thank Reviewer ATTf for the constructive review and valuable suggestions. Following your suggestions, we have conducted additional experiments with two SFT baselines. SFT on agentic traces fine-tunes the model on successful multi-turn trajectories generated by Claude Sonnet 5, while Rejection-sampling SFT fine-tunes the model on successful trajectories sampled from the untrained policy. Both methods use one successful trajectory per training instance. The results on DafnyBench are reported below, which we have incorporated into the revised manuscript.

MethodAnnotation repairCode generation
Untrained policy67.466.1
SFT on reference proofs60.953.2
SFT on agentic traces66.771.8
Rejection-sampling SFT76.076.6
Binary outcome reward75.271.0
Verifier-grounded reward76.793.5

W1. Artificially Weak SFT Baseline: The Supervised Fine-Tuning (SFT) baseline is trained solely on final, polished reference proofs rather than multi-turn problem-solving trajectories. This makes it an uneven comparison against an RL agent that learns from multi-turn interaction and error recovery.

A1. We agree that training on multi-turn trajectories provides a more representative SFT baseline. As shown above, SFT on successful agentic traces improves over SFT on reference proofs, but still substantially underperforms our verifier-grounded RL method, particularly on code generation.


W2. Missing Expert Iteration Comparison: The evaluation lacks a comparison against a highly scalable off-policy method like Expert Iteration or Rejection Sampling. Given the high compute costs of on-policy GRPO, filtering large-scale asynchronous rollouts using the proposed dense reward for iterative SFT might be a highly competitive and faster alternative.

A2. We have added a rejection-sampling SFT baseline that trains on successful agentic trajectories sampled from the untrained policy. While this approach achieves competitive performance on annotation repair (76.0% vs. 76.7%), it falls considerably behind our method on code generation (76.6% vs. 93.5%).


Q1. SFT Data Construction: The current SFT baseline fails to outperform the initial policy because it is trained on final artifacts rather than interactive recovery traces. Could you discuss the feasibility of constructing a "cold-start" dataset of multi-round verifier-guided traces—perhaps by using your dense reward to filter rejection sampling—to train a more representative and robust SFT baseline?

A3. We have constructed successful multi-turn verifier-guided trajectories using both Claude Sonnet 5 and rejection sampling from the untrained policy. As shown above, SFT on these trajectories improves over SFT on reference proofs, but neither approach matches our verifier-grounded RL method.


Q2. Off-Policy vs. On-Policy Efficiency: How do you anticipate the proposed GRPO approach would compare against an Expert Iteration pipeline? Specifically, utilizing large-scale asynchronous rollouts to collect successful or high-partial-credit trajectories, followed by lightweight, iterative offline SFT. Could this off-policy approach be a more compute-efficient way to utilize the dense reward signal?

A4. Our rejection-sampling SFT baseline directly evaluates an off-policy alternative using successful trajectories generated by the base model. Although it underperforms our method, we agree that this approach is more compute-efficient. An interesting direction for future work is to use iterative rejection-sampling SFT as a warm-start stage before RL training to reduce the overall training cost.


Q3. Nature of the Transfer Learning: Regarding the impressive transfer to HumanEval-Verus, is it possible that the observed performance gain primarily stems from the model adapting to the rigid "agentic format" (e.g., learning to emit valid tool tags, handle strictly structured outputs, and avoid unparseable syntax) rather than acquiring deep, language-agnostic verification skills?

A5. Appendix G.3 (Tables 15 and 16) provides evidence that the improvements are not primarily due to learning the agentic format. On HumanEval-Verus code generation, the fraction of unparseable turns actually increases from 51.0% to 70.4%, while the verification rate improves from 17.2% to 36.6%. The fraction of episodes without any submission decreases by only 2.2 percentage points, which cannot explain the 19.4-point improvement. Furthermore, the average number of submissions among successful episodes decreases from 3.25 to 1.50. These results suggest that the transfer gains reflect improved verification capabilities rather than simply better adherence to the agentic interface.


We hope our response addresses your concerns. If you have further concerns, we will be happy to address them during the discussion period.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer ATTf,

Thank you again for your thoughtful review. We wanted to gently remind you that there is one day left in the author-reviewer discussion period, in case you had a chance to review our response and found clarifications helpful. We deeply appreciate the time and effort you’ve already put into this, and are grateful for any further thoughts you may have.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.