- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper propose a valid and meaningful question under current time, which is as more and more scientific discoveries are produced and there are limited human effort to verify them, how can a verifier filter out invalid candidates and avoid rejecting the rare valid ones. The authors treat this problem as a one-sided objective that maximizes the true negative rate under a false negative rate constraint. The author propose a framework OSVerify to solve the problem which divide the whole process into two stages. At first stage, a LLM identify possible error within a work using tree search. At second stage, a verifier verify how each error affect the work. The author also trains the verifier with GRPO to further demonstrate its effectiveness. On multiple math proving benchmark and real-world math research, the proposed method work well.
Critical strengths
Firstly, the proposed question is realistic and meaningful. As more and more AI assisted work appear in research, human experts can no longer verify every work. The author formalize this problem as a one-sided verification problem.
Secondly, the OSVerify method is intuitive and helpful. It contains two part which are error proposer and error validator. The error validator is further trained through RL to make it more effective.
Thirdly, the proposed framework works well on math proving benchmarks and also turns out to be effective under real-world scenario mathematical research.
Critical weaknesses
Firstly, there can be more comparison with competitive baselines. For example, in Appendix B.1, the author mentions "Pessimistic voting". This method has three different variations including "Simple", "Vertical" and "Progressive" version in their original paper. By the definition provided by the author, there is only comparison with the "Simple" version. Comparing with more competitive "Vertical" and "Progressive" version can be helpful.
Secondly, the framework relies on choosing a threshold. On the reported TNR, this is chosen on the evaluation set itself while under real-world scenario, this threshold has to be fixed in advance. Though in Appendix B.4, the author targets this issue and achieves good result, it is only on one benchmark. Providing another benchmark as an example can be helpful.
Questions for authors
-
Could the author compare OSVerify with the "Vertical" and "Progressive" variations of the "Pessimistic voting"? As the progressive version also contains recursively splits the proof, this would be helpful to show how much gain comes from a separate validator.
-
Is it possible to apply held-out threshold selection from Appendix B.4 to H2V and FirstProof-Evaluation? If so, reporting how the TNR and FNR changes compared with selecting threshold on the evaluation set is very helpful.
-
For the conjecture-solving in line 439, it can be helpful to provide how many candidate proofs were generated, how many OSVerify rejected or passed and how much expert time and review was needed. This can provide better understanding of the OSVerify in real-world application.
3 comments
Response to Reviewer uP7t (2/2)
Q3. For the conjecture-solving in line 439, it can be helpful to provide how many candidate proofs were generated, how many OSVerify rejected or passed and how much expert time and review was needed. This can provide better understanding of the OSVerify in real-world application.
Thank you for suggesting additional details on the conjecture-solving workflow and expert review.
For three previously open statements in relative Langlands duality and the Manin constant conjecture, we paired OSVerify with Rethlas [1], an open-source math problem-solving agent, using natural-language proofs without a formal proof checker. Following Rethlas's sequential design, the solver generates one candidate proof per round rather than multiple candidates in parallel. OSVerify reviews each candidate and reports the errors it identifies, which the solver uses to refine or regenerate the proof in the next round. This process continues until OSVerify returns PASS, at which point the proof is forwarded to human experts for final verification.
The table below reports the number of solver–verifier rounds (each reviewing one candidate proof version), the number of proofs passed to human experts, and expert review time.
| Conjecture | Rounds | # proofs passed to the human experts | Expert judgment | Expert review time |
|---|---|---|---|---|
| Flatness of the loop-rotated Plancherel algebra | 2 | 1 | Confirmed correct | 15 min |
| Commutativity of the Plancherel algebra | 3 | 1 | Confirmed correct | 15 min |
| Automorphic commutator relation | 10 | 1 | Confirmed correct | 5 h |
| Manin constant conjecture | 10 | 1 | Confirmed correct | 2 h |
In all four cases, OSVerify forwarded only one proof per conjecture to human experts, and each was confirmed correct on the first expert review. This shows that OSVerify can guide iterative proof refinement while filtering flawed intermediate candidates before expert review. For the Manin constant conjecture, this run reproduced the proof from the statement alone, separately from the manuscript review described in our responses to R-nuya and R-dknW. In the future, we plan to pair OSVerify with larger-scale solving systems that generate multiple candidate proofs in parallel, enabling broader exploration and more effective test-time scaling through verification-guided search.
Kamnitzer's ICM 2022 conjecture. GPT-6 Pro served as the solver, and OSVerify reviewed the proofs separately. In round 1, GPT-6 Pro drafted and reviewed the proof, and Codex with GPT-6-Astra checked it again; OSVerify nevertheless found an error that invalidated the proof of the main theorem. In round 2, GPT-6 Pro revised the proof using OSVerify's feedback. Three further rounds of ChatGPT review and a Codex review all judged the revised proof correct, yet OSVerify found a second proof-breaking error and proposed a fix. After the fix was applied, a human expert verified the corrected proof. This case shows that OSVerify can catch proof-breaking errors in research mathematics that GPT-6 Pro and Codex missed.
Reference
[1] Ju et al. (2026). Automated Conjecture Resolution with Formal Verification.
I thank the author for providing those additional experiments and explanations. Most of my concerns have been solved.
Response to Reviewer uP7t (1/2)
We thank the reviewer for the thoughtful review and for recognizing the motivation for one-sided verification and the potential of the research applications. We address the requested baseline comparisons, held-out calibration, and details of the conjecture-solving workflow below.
Thank you for pointing out these two variants. Progressive Pessimistic Voting is particularly relevant because it also recursively splits the proof.
We use gpt-oss-120b for all three methods and compare attainable TNR under the same empirical FNR caps on the test sets. Vertical splits the proof into n contiguous chunks and checks each chunk, with n ranging from 1 to 7. Progressive starts from the full proof and bisects each chunk in every subsequent round, running for n rounds with n ranging from 1 to 3. Progressive's first round and one-chunk Vertical share the same whole-solution judgment, and both variants reject on the first negative judgment. For OSVerify, we sweep the threshold on the scores from 16 validation votes per proposed error and report the mean ± standard deviation over test replicates, as in Table 1. Due to their cost and the limited rebuttal period, Vertical and Progressive are evaluated from a single run.
Unlike Progressive Pessimistic Voting, which bisects the proof text and rejects on the first negative judgment, OSVerify decomposes the proof along its mathematical argument and independently validates each proposed error before rejection; the gap reflects the combined effect of these two design choices.
Thank you for suggesting this comparison. We additionally evaluate held-out threshold calibration on H2V and FirstProof-Eval, using 24 valid AOBench proofs and 14 valid proofs from First Proof Batch 3 and the Community Experiment, respectively. All calibration problems are disjoint from their corresponding test sets.
We select the least restrictive threshold that meets the target empirical FNR on the calibration set, then apply this fixed threshold to 2,000 test replicates, each subsampling 16 recorded judgments per proof (Voting) or per proposed error (OSVerify). We report mean rates (%) ± standard deviations (percentage points), alongside the TNR obtained by selecting the threshold on the evaluation set (Table 1).
Compared with selecting the threshold on the evaluation set, held-out calibration lowers OSVerify's TNR by only 0.6–2.5 pp while keeping its mean test FNR well below the target. In contrast, Voting's calibrated threshold exceeds the target FNR in some replicates, which is why its calibrated TNR can exceed its eval-set TNR on H2V. Overall, OSVerify achieves a higher mean test TNR and a lower mean test FNR than Voting at both targets on both benchmarks.