- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a training recipe for a reference-free proof generator. The core method is to distill proofs from a teacher model, while the authors show that there are three design choices that can help improve the stability of training, due to the long-context issue in generating proofs.
Critical strengths
-
Training a reference-free proof generator is an interesting problem to explore.
-
Provides extensive training time analysis to show that the three identified design choices have effects on training stability.
Critical weaknesses
-
The reference-free part is done by distilling a stronger teacher’s traces. But this can be a strong assumption + usage + dependence, since it is not surprising that distillation will work?
-
The main contribution of this work is identifying three design choices that improve training stability. However, the length bias in RL has been explored in prior work from more than 1 year ago [1], and the motivation behind the proposed adaptive positive-advantage masking is similar to some existing work too from 1 year ago [2,3]?
-
Being reference-free, the reward is computed by a verifier model that is trained to judge the correctness of the proof y on the problem x, and it is a self-assessment. So how do we know this is a reliable signal? Will self-assessment create bias towards its own generated proof?
-
The models used throughout this paper are mixed and lack clear explanations of why each was chosen. For example, the intro mentioned Nemotron 3 Ultra 550B and DeepSeek-Math-V2, then the experimental setup mentioned Nemotron 3 Nano 30B and Qwen3-4B-Thinking and their distilled versions.
-
In Figure 6, the authors show a comparison with GPT-5.5 as the judge, and it seems that GPT-5.5 is doing slightly better. This raises two questions: (5a) how well GPT-5.5 is when it is being used directly, without any training? If GPT-5.5 is doing well already, then what’s the cost of using it versus training a model using the proposed recipe? (5b) how well is the trained model if it is trained by GPT-5.5’s judgment?
References: [1] https://arxiv.org/abs/2503.20783 [2] https://arxiv.org/abs/2506.01939 [3] https://arxiv.org/abs/2509.03646
Questions for authors
The 5 questions above in weakness section. Additionally,
-
In the abstract and lines 83-86, the authors mentioned “Starting from Nemotron 3 Ultra 550B, we train a proof generator using feedback from a fixed DeepSeek-Math-V2 verifier that receives only the problem and candidate proof. The resulting model surpasses DeepSeek-Math-V2 itself in proof-generation accuracy on downstream evaluation.” It is not clear what “Starting from Nemotron 3 Ultra 550B” means. Which model is being trained here, and which model is generating feedback?
-
Until the end of the introduction, it is not yet clear how to train a verifier without reference. The presentation of the proposed method in the introduction can be improved.
4 comments
Q4: "The models used throughout this paper are mixed and lack clear explanations of why each was chosen."
We deliberately used models at a range of scales, from Qwen3-4B-Thinking through Nemotron 3 Nano 30B to Nemotron 3 Ultra 550B. The smaller models let us iterate quickly on experiments, and the largest one confirmed our combined methodology. At each scale, we chose a relatively new model that performs well for its size. We will add this explanation to Sec. 4.2.
Q5: "(5a) how well GPT-5.5 is when it is being used directly [...]? (5b) how well is the trained model if it is trained by GPT-5.5's judgment?"
There may be a misunderstanding. In Figure 6, GPT-5.5 is the judge of our trained model's proofs, not a prover. All curves in Figure 6 grade the same Nemotron 3 Ultra proofs.
(5a) Using GPT-5.5 directly as a proof generator is certainly possible. Our goal, however, is to study proof generation, specifically how to train a proof generator with a verifier's feedback, rather than to compete with the best closed-source models.
(5b) Training with GPT-5.5 judgments is prohibitively expensive. Our Nemotron 3 Ultra RL run needed about 140K graded rollouts, and judging each proof requires a long chain of thought (DeepSeek-Math-V2's judgments average 17–21K tokens).
Q6: "Which model is being trained here, and which model is generating feedback?"
The model being trained is Nemotron 3 Ultra 550B, and the feedback comes from a frozen DeepSeek-Math-V2 verifier. We will state this explicitly in the abstract and introduction.
Q7: "Until the end of the introduction, it is not yet clear how to train a verifier without reference."
The focus of our paper is training a proof generator against a verifier's feedback. Training a verifier is outside the scope of our paper. It is, however, a very interesting question and a valuable research direction.
We hope these clarifications address the reviewer's concerns, and we would be glad to answer any further questions.
reply to author response
Thanks to the authors for the detailed response and clarifications. The rebuttal addresses some of my original concerns about distillation and the use of a separate frozen verifier. However, I still have two important concerns.
-
Masking low-probability tokens to control entropy is not new, and the RL setup appears closely related to existing work [1, 2, 3]. Clear clarification is needed on what distinguishes this submission from past work, both methodologically and empirically.
-
While the improvement from 45.0% to 57.6% is encouraging, it is still unclear how much can be attributed to the three proposed design choices rather than RL with a strong pretrained model and verifier. In particular, stronger comparisons with simpler alternatives (e.g., random masking at the same rate or standard entropy regularization) would help establish the necessity of Ada-POS.
Overall, I appreciate the authors' efforts but there are still questions/unclarity about methodological novelty and the attribution of the gains, so I will maintain my original rating. Thanks!
[1] https://arxiv.org/abs/2602.15620 [2] https://arxiv.org/abs/2505.12929 [3] https://arxiv.org/abs/2609.10712
We thank the reviewer for engaging with our rebuttal and for bringing these three works to our attention.
Q8: "Masking low-probability tokens to control entropy is not new, and the RL setup appears closely related to existing work [1, 2, 3]."
[1] and [3] are very recent, concurrent works. [2] does not mask out low-probability tokens. It divides tokens into low-probability and high-probability groups, typically with a threshold of 0.5, and performs two separate optimization updates, one on each group, which requires two backward passes. Our method instead focuses on entropy control, masks a very small number of tokens (less than 0.01% of all training tokens), and requires only a single backward pass.
Q9: "[...] stronger comparisons with simpler alternatives (e.g., random masking at the same rate or standard entropy regularization) would help establish the necessity of Ada-POS."
Random masking is equivalent to the original policy-gradient loss. Eq. 2 of the paper defines our policy-gradient estimator , where is the importance-weighting rule (TIS in our runs). It is the expected value, over problems and groups of sampled proofs , of a sum of per-token terms
so that . A random mask drops each token independently with probability , so it equals 0 with probability and 1 otherwise. Since the mask is independent of the sampled proofs, , and
Random masking therefore recovers the original objective, only rescaled by the constant factor . The Ada-POS mask , in contrast, depends on the token's advantage and probability, so it changes the direction of the update.
We would be glad to answer any further questions.
We thank the reviewer for the detailed feedback.
Q1: "The reference-free part is done by distilling a stronger teacher's traces [...]"
No. Our main result involves no distillation, and all of its gains come from RL. Nemotron 3 Ultra 550B starts from its public checkpoint and is trained only with rewards from the frozen verifier. Its accuracy on ProofBench25 rises from 45.0% to 57.6%, while the verifier itself, DeepSeek-Math-V2, reaches 46.9% as a prover.
We use distillation only for our smaller proxy models. Small models are often weak at proof generation, and our setup needs them to both generate and verify proofs, so we first bootstrap them for this task.
Q2: "the length bias in RL has been explored in prior work [...] [1], and the motivation behind the proposed adaptive positive-advantage masking is similar to [...] [2,3]?"
Our findings differ from [1], and our motivation and conclusions differ from those of [2] and [3].
[1] (Dr. GRPO, already cited in the paper) concludes that the per-sequence normalization makes reasoning length grow excessively. Yet reasoning length often grows during RL both with and without such normalization (e.g., DeepSeek-R1 and DAPO). What we note instead is the length of the proof, which is what the verifier sees and, for Nemotron 3 Ultra, is less than 5% of the total generated tokens.
[2] selects tokens based on their entropy. Ada-POS instead masks out positive-advantage tokens based on their absolute probability, in order to control entropy. In fact, Ada-POS masks out less than 0.01% of all training tokens, yet this is enough to keep entropy near its target (Fig. 3).
As we understand it, [3] is an advantage-shaping method that aims to improve exploration, whereas Ada-POS controls entropy to prevent it from exploding. We will cite and discuss both [2] and [3].
Q3: "[...] it is a self-assessment. So how do we know this is a reliable signal? Will self-assessment create bias towards its own generated proof?"
There is no self-assessment in our reward, and the verifier never grades its own proofs during training. The reward comes from a separate, frozen DeepSeek-Math-V2 that grades Nemotron 3 Ultra's proofs. We removed DeepSeek-Math-V2's self-assessment reward term because it produced a "lazy" policy (Sec. 4, App. A).
GPT-5.5, an independent judge never used in training, agrees with the improvement. From the beginning to the end of RL, its accuracy rises from 45.0% to 57.6%.
If your question is about the reliability of DeepSeek-Math-V2, it is consistent with itself. When it graded the same proof twice, its two grades split between 0 and ≥0.5 for only 2.54% of 136,826 proofs.