acceptodds
Under review as a conference paper at ICLR 2027

Training Mathematical Proof Generators via Reference Free Generative Verifiers

Abstract

Mathematical reasoning goes beyond finding correct answers to constructing rigorous proofs. Recent efforts have used reinforcement learning with verifier feedback to improve language models' ability to generate such proofs. However, the long reasoning traces required for generation and verification make reinforcement learning in this setting difficult to stabilize. Through ablations on smaller models, we identify three choices that improve training stability: sequence-level rather than token-level loss aggregation, large batches, and adaptive positive-advantage masking. This masking method controls policy entropy by selectively suppressing updates to low-probability tokens with positive advantage. Starting from Nemotron 3 Ultra, we train a proof generator using feedback from a fixed DeepSeek-Math-V2 verifier that receives only the problem and candidate proof. The resulting model surpasses DeepSeek-Math-V2 itself in proof-generation accuracy on downstream evaluation. The gains persist under an independent judge and reference-aware evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qKXnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a training recipe for a reference-free proof generator. The core method is to distill proofs from a teacher model, while the authors show that there are three design choices that can help improve the stability of training, due to the long-context issue in generating proofs.

Critical strengths
  1. Training a reference-free proof generator is an interesting problem to explore.

  2. Provides extensive training time analysis to show that the three identified design choices have effects on training stability.

Critical weaknesses
  1. The reference-free part is done by distilling a stronger teacher’s traces. But this can be a strong assumption + usage + dependence, since it is not surprising that distillation will work?

  2. The main contribution of this work is identifying three design choices that improve training stability. However, the length bias in RL has been explored in prior work from more than 1 year ago [1], and the motivation behind the proposed adaptive positive-advantage masking is similar to some existing work too from 1 year ago [2,3]?

  3. Being reference-free, the reward is computed by a verifier model that is trained to judge the correctness of the proof y on the problem x, and it is a self-assessment. So how do we know this is a reliable signal? Will self-assessment create bias towards its own generated proof?

  4. The models used throughout this paper are mixed and lack clear explanations of why each was chosen. For example, the intro mentioned Nemotron 3 Ultra 550B and DeepSeek-Math-V2, then the experimental setup mentioned Nemotron 3 Nano 30B and Qwen3-4B-Thinking and their distilled versions.

  5. In Figure 6, the authors show a comparison with GPT-5.5 as the judge, and it seems that GPT-5.5 is doing slightly better. This raises two questions: (5a) how well GPT-5.5 is when it is being used directly, without any training? If GPT-5.5 is doing well already, then what’s the cost of using it versus training a model using the proposed recipe? (5b) how well is the trained model if it is trained by GPT-5.5’s judgment?

References: [1] https://arxiv.org/abs/2503.20783 [2] https://arxiv.org/abs/2506.01939 [3] https://arxiv.org/abs/2509.03646

Questions for authors

The 5 questions above in weakness section. Additionally,

  1. In the abstract and lines 83-86, the authors mentioned “Starting from Nemotron 3 Ultra 550B, we train a proof generator using feedback from a fixed DeepSeek-Math-V2 verifier that receives only the problem and candidate proof. The resulting model surpasses DeepSeek-Math-V2 itself in proof-generation accuracy on downstream evaluation.” It is not clear what “Starting from Nemotron 3 Ultra 550B” means. Which model is being trained here, and which model is generating feedback?

  2. Until the end of the introduction, it is not yet clear how to train a verifier without reference. The presentation of the proposed method in the introduction can be improved.

4 comments

We thank the reviewer for the detailed feedback.

Q1: "The reference-free part is done by distilling a stronger teacher's traces [...]"

No. Our main result involves no distillation, and all of its gains come from RL. Nemotron 3 Ultra 550B starts from its public checkpoint and is trained only with rewards from the frozen verifier. Its accuracy on ProofBench25 rises from 45.0% to 57.6%, while the verifier itself, DeepSeek-Math-V2, reaches 46.9% as a prover.

We use distillation only for our smaller proxy models. Small models are often weak at proof generation, and our setup needs them to both generate and verify proofs, so we first bootstrap them for this task.

Q2: "the length bias in RL has been explored in prior work [...] [1], and the motivation behind the proposed adaptive positive-advantage masking is similar to [...] [2,3]?"

Our findings differ from [1], and our motivation and conclusions differ from those of [2] and [3].

[1] (Dr. GRPO, already cited in the paper) concludes that the per-sequence normalization makes reasoning length grow excessively. Yet reasoning length often grows during RL both with and without such normalization (e.g., DeepSeek-R1 and DAPO). What we note instead is the length of the proof, which is what the verifier sees and, for Nemotron 3 Ultra, is less than 5% of the total generated tokens.

[2] selects tokens based on their entropy. Ada-POS instead masks out positive-advantage tokens based on their absolute probability, in order to control entropy. In fact, Ada-POS masks out less than 0.01% of all training tokens, yet this is enough to keep entropy near its target (Fig. 3).

As we understand it, [3] is an advantage-shaping method that aims to improve exploration, whereas Ada-POS controls entropy to prevent it from exploding. We will cite and discuss both [2] and [3].

Q3: "[...] it is a self-assessment. So how do we know this is a reliable signal? Will self-assessment create bias towards its own generated proof?"

There is no self-assessment in our reward, and the verifier never grades its own proofs during training. The reward comes from a separate, frozen DeepSeek-Math-V2 that grades Nemotron 3 Ultra's proofs. We removed DeepSeek-Math-V2's self-assessment reward term because it produced a "lazy" policy (Sec. 4, App. A).

GPT-5.5, an independent judge never used in training, agrees with the improvement. From the beginning to the end of RL, its accuracy rises from 45.0% to 57.6%.

If your question is about the reliability of DeepSeek-Math-V2, it is consistent with itself. When it graded the same proof twice, its two grades split between 0 and ≥0.5\geq 0.5 for only 2.54% of 136,826 proofs.

Q4: "The models used throughout this paper are mixed and lack clear explanations of why each was chosen."

We deliberately used models at a range of scales, from Qwen3-4B-Thinking through Nemotron 3 Nano 30B to Nemotron 3 Ultra 550B. The smaller models let us iterate quickly on experiments, and the largest one confirmed our combined methodology. At each scale, we chose a relatively new model that performs well for its size. We will add this explanation to Sec. 4.2.

Q5: "(5a) how well GPT-5.5 is when it is being used directly [...]? (5b) how well is the trained model if it is trained by GPT-5.5's judgment?"

There may be a misunderstanding. In Figure 6, GPT-5.5 is the judge of our trained model's proofs, not a prover. All curves in Figure 6 grade the same Nemotron 3 Ultra proofs.

(5a) Using GPT-5.5 directly as a proof generator is certainly possible. Our goal, however, is to study proof generation, specifically how to train a proof generator with a verifier's feedback, rather than to compete with the best closed-source models.

(5b) Training with GPT-5.5 judgments is prohibitively expensive. Our Nemotron 3 Ultra RL run needed about 140K graded rollouts, and judging each proof requires a long chain of thought (DeepSeek-Math-V2's judgments average 17–21K tokens).

Q6: "Which model is being trained here, and which model is generating feedback?"

The model being trained is Nemotron 3 Ultra 550B, and the feedback comes from a frozen DeepSeek-Math-V2 verifier. We will state this explicitly in the abstract and introduction.

Q7: "Until the end of the introduction, it is not yet clear how to train a verifier without reference."

The focus of our paper is training a proof generator against a verifier's feedback. Training a verifier is outside the scope of our paper. It is, however, a very interesting question and a valuable research direction.

We hope these clarifications address the reviewer's concerns, and we would be glad to answer any further questions.

Reviewer_qKXnOpenReview ↗

reply to author response

Thanks to the authors for the detailed response and clarifications. The rebuttal addresses some of my original concerns about distillation and the use of a separate frozen verifier. However, I still have two important concerns.

  1. Masking low-probability tokens to control entropy is not new, and the RL setup appears closely related to existing work [1, 2, 3]. Clear clarification is needed on what distinguishes this submission from past work, both methodologically and empirically.

  2. While the improvement from 45.0% to 57.6% is encouraging, it is still unclear how much can be attributed to the three proposed design choices rather than RL with a strong pretrained model and verifier. In particular, stronger comparisons with simpler alternatives (e.g., random masking at the same rate or standard entropy regularization) would help establish the necessity of Ada-POS.

Overall, I appreciate the authors' efforts but there are still questions/unclarity about methodological novelty and the attribution of the gains, so I will maintain my original rating. Thanks!

[1] https://arxiv.org/abs/2602.15620 [2] https://arxiv.org/abs/2505.12929 [3] https://arxiv.org/abs/2609.10712

We thank the reviewer for engaging with our rebuttal and for bringing these three works to our attention.

Q8: "Masking low-probability tokens to control entropy is not new, and the RL setup appears closely related to existing work [1, 2, 3]."

[1] and [3] are very recent, concurrent works. [2] does not mask out low-probability tokens. It divides tokens into low-probability and high-probability groups, typically with a threshold of 0.5, and performs two separate optimization updates, one on each group, which requires two backward passes. Our method instead focuses on entropy control, masks a very small number of tokens (less than 0.01% of all training tokens), and requires only a single backward pass.

Q9: "[...] stronger comparisons with simpler alternatives (e.g., random masking at the same rate or standard entropy regularization) would help establish the necessity of Ada-POS."

Random masking is equivalent to the original policy-gradient loss. Eq. 2 of the paper defines our policy-gradient estimator gω(θ)g_\omega(\theta), where ω\omega is the importance-weighting rule (TIS in our runs). It is the expected value, over problems xx and groups of sampled proofs {yi}\lbrace y_i \rbrace, of a sum of per-token terms

fi,t=1GTiω(ρi,t,A^i)A^i∇θlog⁡πtrain(yi,t∣x,yi,<t),f_{i,t} = \frac{1}{G T_i} \omega(\rho_{i,t}, \widehat A_i) \widehat A_i \nabla_\theta \log \pi_{\mathrm{train}}(y_{i,t} \mid x, y_{i,\lt t}),

so that gω(θ)=E[∑i,tfi,t]g_\omega(\theta) = \mathbb{E}\big[\sum_{i,t} f_{i,t}\big]. A random mask m~i,t\tilde m_{i,t} drops each token independently with probability qq, so it equals 0 with probability qq and 1 otherwise. Since the mask is independent of the sampled proofs, E[m~i,t∣x,{yi}]=1−q\mathbb{E}[\tilde m_{i,t} \mid x, \lbrace y_i \rbrace] = 1-q, and

E[∑i,tm~i,tfi,t]=E[∑i,tE[m~i,t∣x,{yi}]fi,t]=(1−q)E[∑i,tfi,t]=(1−q)gω(θ).\mathbb{E}\Big[\sum_{i,t} \tilde m_{i,t} f_{i,t}\Big] = \mathbb{E}\Big[\sum_{i,t} \mathbb{E}[\tilde m_{i,t} \mid x, \lbrace y_i \rbrace] f_{i,t}\Big] = (1-q) \mathbb{E}\Big[\sum_{i,t} f_{i,t}\Big] = (1-q) g_\omega(\theta).

Random masking therefore recovers the original objective, only rescaled by the constant factor 1−q1-q. The Ada-POS mask mi,tm_{i,t}, in contrast, depends on the token's advantage and probability, so it changes the direction of the update.

We would be glad to answer any further questions.

Reviewer_jbLYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning for long-context mathematical proof generation using a fixed, reference-free generative verifier. The authors identify several challenges in stabilizing GRPO with long trajectories and propose three improvements: sequence-level loss aggregation, larger batch sizes, and adaptive positive-advantage masking for entropy control. Experiments demonstrate improved training stability and proof-generation performance, with the trained generator surpassing the proof-generation accuracy of its verifier.

Critical strengths
  1. Practical challenges in long-context RL. The paper studies an important and challenging setting where both proof generation and verification require extremely long reasoning trajectories. The authors identify several training instabilities and provide practical techniques to address them.

  2. Systematic empirical analysis. The paper investigates multiple factors affecting training stability, including loss aggregation, batch size, and entropy control. The experiments provide useful empirical insights into training RL models with long responses.

  3. Strong empirical results. The authors demonstrate improvements in proof-generation performance using a fixed verifier. Notably, the resulting generator surpasses the verifier's own proof-generation accuracy, and the improvements persist under an independent evaluation model.

Critical weaknesses
  1. Limited comparison with existing RL stabilization methods. Although the paper provides several ablation studies, the proposed methods are not sufficiently compared against closely related techniques. In particular, Ada-POS is inspired by the adaptive entropy controller in MAI-Thinking-1, but the paper does not directly compare their effectiveness. Similarly, the benefits of sequence-level loss aggregation could be further examined against existing sequence-level optimization methods such as GSPO. These comparisons would help clarify the effectiveness and novelty of the proposed approach.

  2. Limited generalization beyond mathematical proof generation. Although the paper primarily targets mathematical proof generation, its proposed techniques address general challenges in long-context RL, such as entropy instability and off-policy training. However, the experiments are restricted to mathematical proof-generation tasks. Evaluating the proposed methods on other long-response reasoning tasks, such as mathematical problem solving or code generation, would help establish whether the findings generalize beyond this specific setting.

Questions for authors

My questions are the same as those raised in the Weaknesses section.

1 comment

We thank the reviewer for recognizing the practical challenges we address, our systematic analysis, and that the trained generator surpasses its verifier's own proof-generation accuracy.

Q1: "Ada-POS is inspired by the adaptive entropy controller in MAI-Thinking-1, but the paper does not directly compare their effectiveness."

We thank the reviewer for this suggestion. We ran this comparison on Nemotron 3 Nano.

ProofBench25 accuracy (%), GPT-5.5 judgeInitStep 50Step 100Step 150Step 200
TIS + Ada-POS (ours)31.734.033.735.938.2
MAI-Thinking-1's controller31.733.832.836.434.9

The two approaches perform comparably. Ada-POS, however, is applicable to any importance-sampling method, while MAI-Thinking-1's controller is tailored to PPO-like clipping. We will add the full per-step comparison to the appendix.

Q2: "[...] the proposed methods are not sufficiently compared against closely related techniques. [...] the benefits of sequence-level loss aggregation could be further examined against existing sequence-level optimization methods such as GSPO."

In all of our settings, we already adopt a sequence-level mask similar to GSPO's (App. D). Regarding other stabilization methods, we experimented with IcePop [1] using two settings of its ratio bounds, [0.8, 1.3] and [0.5, 2.0]. Neither showed any advantage over TIS, so we did not include IcePop in the paper. We will add these results to the appendix.

Q3: "Limited generalization beyond mathematical proof generation."

The main focus of our paper is mathematical proof generation. While we do not see any obstacles to generalizing our findings to other domains, each domain requires separate investigation and tuning, which is outside the scope of our paper.

We are happy to answer further questions.

[1] Ling Team, 2025 (Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model, arXiv:2510.18855).

Reviewer_Hv8xOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reinforcement learning for mathematical proof generation using a fixed, reference-free generative verifier. It investigates training instability arising from long rollouts and expensive verification, and proposes a recipe combining sequence-level loss aggregation, larger batches, and adaptive positive-advantage masking (Ada-POS). Validation includes smaller-model ablations and training Nemotron 3 Ultra with DeepSeek-Math-V2 as the verifier. The authors report improved proof-generation performance, including gains under alternative judges and reference-aware evaluation.

Critical strengths

Strong experimental formulation with a stationary verifier. Holding the verifier fixed is an important scientific design choice: it separates improvements in the generator from changes in the reward model. This makes the contribution of generator training easier to assess and provides a useful setting for investigating what a generator can learn from a fixed verifier.

Informative investigation of a meaningful systems problem. Very long rollouts combined with expensive generative verification create substantial challenges for asynchronous RL. The failure analyses in Figures 2–4 are particularly relevant, illustrating how loss aggregation, entropy control, and batch configuration affect training. Ada-POS is simple to understand and well explained, making the proposed intervention accessible and potentially useful to subsequent work.

Critical weaknesses

The evaluation does not sufficiently establish complete-proof correctness. Lines 457–458 define binary accuracy by counting both partially correct proofs, scored 0.5, and fully correct proofs, scored 1, as correct. This metric measures attainment of at least partial credit rather than full correctness. Although evaluation with additional judges is valuable, different models may share preferences for presentation and verbosity rather than independently establish mathematical validity. This is particularly relevant because average final-proof length increases from approximately 4,699 to 7,584 characters (lines 465–468). Additional detail may improve proofs, but it may also influence automated grading independently of correctness. Reporting the full-credit rate separately and providing blinded human validation would substantially strengthen the central claim.

Insufficient documentation of training data prevents assessment of contamination. Section 4.2 explains how distillation traces are generated, but provides insufficient detail about the underlying problem sources, dataset sizes, competition years, filtering, and overlap with evaluation problems. The RL training corpus also needs clearer documentation. Without this information and explicit deduplication checks, I cannot adequately assess whether the reported benchmark gains reflect generalization. This is a documentation and validation concern, not evidence that contamination occurred.

Questions for authors

Metric clarification and reanalysis: What are the full-credit rates when only verifier scores of 1 count as correct? Does the improvement over initialization—and the comparison with DeepSeek-Math-V2—remain under this definition? Please also identify the grading protocol used in Figure 1.

Evaluation validation: Do you have blinded human evaluations or controlled analyses that distinguish improvements in mathematical validity from changes in proof length and presentation? If available, these results would materially strengthen my assessment.

Data clarification: What are the sources, sizes, and competition-year coverage of the distillation and RL datasets? What exact and near-duplicate checks were performed against every component of ProofBench25, and were any overlaps found?

3 comments

Response to Reviewer Hv8x

We thank the reviewer for the careful review, and we are glad they value holding the verifier fixed and found the failure analyses useful.

Q1: "What are the full-credit rates when only verifier scores of 1 count as correct? Does the improvement over initialization—and the comparison with DeepSeek-Math-V2—remain under this definition? Please also identify the grading protocol used in Figure 1."

Yes to both. Under full credit, where only a score of 1 counts, the improvement over initialization is even larger. With DeepSeek-Math-V2 as the judge, accuracy rises from 20.9% to 40.6% from the beginning to the end of RL (+19.7 pp, versus +13.2 pp under the paper's score ≥0.5\geq 0.5), and our model stays ahead of DeepSeek-Math-V2's own proofs (33.4%). GPT-5.5 shows the same pattern.

ProofBench25 accuracy (%)DeepSeek-Math-V2's proofsOurs, before RLOurs, after RL
DeepSeek-Math-V2 judge, full credit33.420.940.6
GPT-5.5 judge, full credit41.534.451.1
GPT-5.5 judge, Figure 1 protocol46.945.057.6

Figure 1 uses GPT-5.5 with the Open Proof Corpus (OPC) prompt, plus a reference proof when one exists. It gives a yes/no verdict, counting a proof as correct only if it would earn at least 5 of 7 points. We will state this protocol in the Figure 1 caption and add the full-credit results to Figure 6.

Q2: "Do you have blinded human evaluations or controlled analyses that distinguish improvements in mathematical validity from changes in proof length and presentation?"

No. At this scale, blinded human evaluation is impractical. It is standard practice to use LLM judges as a proxy for human graders, as in prior work [1, 2, 3]. These works report very high agreement with human graders, particularly without test-time scaling, as in our evaluation. In our case, GPT-5.5, an independent judge never used in training, agrees with the improvement, with accuracy rising from 45.0% to 57.6% from the beginning to the end of RL.

Q3: "What are the sources, sizes, and competition-year coverage of the distillation and RL datasets? What exact and near-duplicate checks were performed [...]?"

All problems in our SFT and RL data come from the Art of Problem Solving (AoPS) forum, up to a knowledge cutoff in 2024. ProofBench25 consists of 2025 competition problems and IMO-ProofBench [2], which was released after 2024 and whose authors modified the statements of earlier problems. There is therefore a temporal separation between our training data and the evaluation problems. Given this separation, we did not perform near-duplicate checks.

Our SFT data covers 19,925 problems. For RL, we use 65,882, 35,444 and 9,596 problems for Qwen3-4B, Nemotron 3 Nano and Nemotron 3 Ultra, respectively. For each model, we chose problems of appropriate difficulty. We will add these details to the paper.

To make sure that the modified problems from IMO-ProofBench do not inflate the gains, we also compute accuracy on the subset of problems that are strictly from 2025 competitions or novel (101 of the 133). On this subset, accuracy rises from 39.4% to 53.5% from the beginning to the end of RL.

We would be glad to answer any further questions.

[1] Dekoninck et al., 2025 (The Open Proof Corpus). [2] Luong et al., 2025 (IMO-ProofBench). [3] Mahdavi et al., 2026 (Scaling Generative Verifiers, ICML).

Reviewer_Hv8xOpenReview ↗

Follow-up on Author Response and Additional Experiments

Thank you for the clarifications and additional results. The full-credit evaluations address my concern about the permissive binary metric.

My remaining concern is that agreement across LLM judges does not directly distinguish improved mathematical validity from changes in presentation and proof length. My request concerned a limited human audit or controlled analysis, rather than exhaustive evaluation. If neither is feasible, please explicitly acknowledge this limitation.

Could you also clarify whether the full-credit improvement and superiority over DeepSeek-Math-V2 persist on the 101-problem restricted subset? Please incorporate the new results and grading definitions into the revised manuscript. I would like to hold my current scores.

We thank the reviewer for the follow-up and are glad that the full-credit results address the concern about the binary metric.

Q4: "My remaining concern is that agreement across LLM judges does not directly distinguish improved mathematical validity from changes in presentation and proof length. [...] If neither is feasible, please explicitly acknowledge this limitation."

Evaluating proof generators with LLM judges is not specific to our work. It is common practice in recent work [1, 2]. Nevertheless, the revised paper now explicitly acknowledges the lack of a human evaluation as a limitation.

Q5: "Could you also clarify whether the full-credit improvement and superiority over DeepSeek-Math-V2 persist on the 101-problem restricted subset?"

Yes, both persist. On the 101-problem subset, both judges report a full-credit gain of at least 17 pp from the beginning to the end of RL. Our final model also stays ahead of DeepSeek-Math-V2's own proofs, by 6.7 pp with the DeepSeek-Math-V2 judge and 9.1 pp with the GPT-5.5 judge.

Full credit on the 101-problem subset (%)DeepSeek-Math-V2's proofsOurs, before RLOurs, after RLImprovement (pp)
DeepSeek-Math-V2 judge29.315.436.0+20.6
GPT-5.5 judge36.828.945.9+17.0

As requested, we have incorporated everything discussed in the rebuttal into the revised paper, with all changes highlighted in blue. We would be glad to answer any further questions.

[1] Shao et al., 2025 (DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning). [2] Qu et al., 2026 (QED-Nano: Teaching a Tiny Model to Prove Hard Theorems).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.