acceptodds
Under review as a conference paper at ICLR 2027

One-Sided Verification Under Limited Expert Review

Abstract

AI has become increasingly helpful in scientific discovery, yet human expert review remains essential for those problems without reliable automated evaluation, posing a fundamental challenge as candidate generation scales beyond available review capacity. This paper studies verification under limited human effort, where rare valid solutions need to be prioritized while invalid candidates should be filtered out to reduce human workload. We adopt the one-sided verification objective that maximizes the rejection rate of invalid solutions (TNR) under a constraint on the false-rejection rate of valid ones (FNR), making human review cost an explicit term to minimize. Drawing on the intuition that verifying a specific error can be easier than discovering one, we introduce \method, an algorithmic framework that decouples error proposal from error validation. The error proposer searches for errors in solutions across different granularities by hierarchical decomposition, while the validator distinguishes substantial errors from minor or hallucinated ones. Experiments on competition- and research-level benchmarks show that OSVerify reduces the number of human reviews of invalid solutions by 46% on average compared to voting baselines under the same empirical low FNR constraint. Our post-training of the validator further improves the reduction to 57%. As high-impact real-world use cases, we apply OSVerify to identify errors in published research papers that are subsequently confirmed by the original authors and help resolve three Langlands open conjectures that are confirmed by experts, demonstrating its utility to facilitate human-AI research collaboration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Reference-free Verification, Scientific reasoning, Selective prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uP7tOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper propose a valid and meaningful question under current time, which is as more and more scientific discoveries are produced and there are limited human effort to verify them, how can a verifier filter out invalid candidates and avoid rejecting the rare valid ones. The authors treat this problem as a one-sided objective that maximizes the true negative rate under a false negative rate constraint. The author propose a framework OSVerify to solve the problem which divide the whole process into two stages. At first stage, a LLM identify possible error within a work using tree search. At second stage, a verifier verify how each error affect the work. The author also trains the verifier with GRPO to further demonstrate its effectiveness. On multiple math proving benchmark and real-world math research, the proposed method work well.

Critical strengths

Firstly, the proposed question is realistic and meaningful. As more and more AI assisted work appear in research, human experts can no longer verify every work. The author formalize this problem as a one-sided verification problem.

Secondly, the OSVerify method is intuitive and helpful. It contains two part which are error proposer and error validator. The error validator is further trained through RL to make it more effective.

Thirdly, the proposed framework works well on math proving benchmarks and also turns out to be effective under real-world scenario mathematical research.

Critical weaknesses

Firstly, there can be more comparison with competitive baselines. For example, in Appendix B.1, the author mentions "Pessimistic voting". This method has three different variations including "Simple", "Vertical" and "Progressive" version in their original paper. By the definition provided by the author, there is only comparison with the "Simple" version. Comparing with more competitive "Vertical" and "Progressive" version can be helpful.

Secondly, the framework relies on choosing a threshold. On the reported TNR, this is chosen on the evaluation set itself while under real-world scenario, this threshold has to be fixed in advance. Though in Appendix B.4, the author targets this issue and achieves good result, it is only on one benchmark. Providing another benchmark as an example can be helpful.

Questions for authors
  1. Could the author compare OSVerify with the "Vertical" and "Progressive" variations of the "Pessimistic voting"? As the progressive version also contains recursively splits the proof, this would be helpful to show how much gain comes from a separate validator.

  2. Is it possible to apply held-out threshold selection from Appendix B.4 to H2V and FirstProof-Evaluation? If so, reporting how the TNR and FNR changes compared with selecting threshold on the evaluation set is very helpful.

  3. For the conjecture-solving in line 439, it can be helpful to provide how many candidate proofs were generated, how many OSVerify rejected or passed and how much expert time and review was needed. This can provide better understanding of the OSVerify in real-world application.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uP7t (1/2)

We thank the reviewer for the thoughtful review and for recognizing the motivation for one-sided verification and the potential of the research applications. We address the requested baseline comparisons, held-out calibration, and details of the conjecture-solving workflow below.

Q1. Could the author compare OSVerify with the "Vertical" and "Progressive" variations of the "Pessimistic voting"? As the progressive version also contains recursively splits the proof, this would be helpful to show how much gain comes from a separate validator.

Thank you for pointing out these two variants. Progressive Pessimistic Voting is particularly relevant because it also recursively splits the proof.

We use gpt-oss-120b for all three methods and compare attainable TNR under the same empirical FNR caps on the test sets. Vertical splits the proof into n contiguous chunks and checks each chunk, with n ranging from 1 to 7. Progressive starts from the full proof and bisects each chunk in every subsequent round, running for n rounds with n ranging from 1 to 3. Progressive's first round and one-chunk Vertical share the same whole-solution judgment, and both variants reject on the first negative judgment. For OSVerify, we sweep the threshold on the scores from 16 validation votes per proposed error and report the mean ± standard deviation over test replicates, as in Table 1. Due to their cost and the limited rebuttal period, Vertical and Progressive are evaluated from a single run.

BenchmarkFNR capVertical PessimisticProgressive PessimisticOSVerify
Putnam1005%0.00.070.4 ± 14.3
Putnam10010%0.00.077.9 ± 3.3
H2V5%77.952.884.7 ± 0.6
H2V10%77.980.484.7 ± 0.5
FirstProof-Eval5%0.00.058.0 ± 3.1
FirstProof-Eval10%42.442.460.8 ± 1.3

Unlike Progressive Pessimistic Voting, which bisects the proof text and rejects on the first negative judgment, OSVerify decomposes the proof along its mathematical argument and independently validates each proposed error before rejection; the gap reflects the combined effect of these two design choices.

Q2. Is it possible to apply held-out threshold selection from Appendix B.4 to H2V and FirstProof-Evaluation? If so, reporting how the TNR and FNR changes compared with selecting threshold on the evaluation set is very helpful.

Thank you for suggesting this comparison. We additionally evaluate held-out threshold calibration on H2V and FirstProof-Eval, using 24 valid AOBench proofs and 14 valid proofs from First Proof Batch 3 and the Community Experiment, respectively. All calibration problems are disjoint from their corresponding test sets.

We select the least restrictive threshold that meets the target empirical FNR on the calibration set, then apply this fixed threshold to 2,000 test replicates, each subsampling 16 recorded judgments per proof (Voting) or per proposed error (OSVerify). We report mean rates (%) ± standard deviations (percentage points), alongside the TNR obtained by selecting the threshold on the evaluation set (Table 1).

Test benchmarkTarget FNREval-set threshold (Table 1): Voting / OSVerify TNRHeld-out calibration: Voting test FNR / TNRHeld-out calibration: OSVerify test FNR / TNR
H2V5%69.2 / 84.74.0 ± 1.7 / 71.2 ± 1.50.0 ± 0.0 / 83.3 ± 0.5
H2V10%75.6 / 84.77.7 ± 2.4 / 78.3 ± 1.50.0 ± 0.0 / 84.1 ± 0.5
FirstProof-Eval5%35.9 / 58.03.9 ± 1.7 / 36.6 ± 1.11.2 ± 1.6 / 55.9 ± 1.7
FirstProof-Eval10%48.8 / 60.86.3 ± 1.2 / 45.0 ± 1.72.8 ± 2.0 / 58.3 ± 1.4

Compared with selecting the threshold on the evaluation set, held-out calibration lowers OSVerify's TNR by only 0.6–2.5 pp while keeping its mean test FNR well below the target. In contrast, Voting's calibrated threshold exceeds the target FNR in some replicates, which is why its calibrated TNR can exceed its eval-set TNR on H2V. Overall, OSVerify achieves a higher mean test TNR and a lower mean test FNR than Voting at both targets on both benchmarks.

AuthorsOpenReview ↗

Response to Reviewer uP7t (2/2)

Q3. For the conjecture-solving in line 439, it can be helpful to provide how many candidate proofs were generated, how many OSVerify rejected or passed and how much expert time and review was needed. This can provide better understanding of the OSVerify in real-world application.

Thank you for suggesting additional details on the conjecture-solving workflow and expert review.

For three previously open statements in relative Langlands duality and the Manin constant conjecture, we paired OSVerify with Rethlas [1], an open-source math problem-solving agent, using natural-language proofs without a formal proof checker. Following Rethlas's sequential design, the solver generates one candidate proof per round rather than multiple candidates in parallel. OSVerify reviews each candidate and reports the errors it identifies, which the solver uses to refine or regenerate the proof in the next round. This process continues until OSVerify returns PASS, at which point the proof is forwarded to human experts for final verification.

The table below reports the number of solver–verifier rounds (each reviewing one candidate proof version), the number of proofs passed to human experts, and expert review time.

ConjectureRounds# proofs passed to the human expertsExpert judgmentExpert review time
Flatness of the loop-rotated Plancherel algebra21Confirmed correct15 min
Commutativity of the Plancherel algebra31Confirmed correct15 min
Automorphic commutator relation101Confirmed correct5 h
Manin constant conjecture101Confirmed correct2 h

In all four cases, OSVerify forwarded only one proof per conjecture to human experts, and each was confirmed correct on the first expert review. This shows that OSVerify can guide iterative proof refinement while filtering flawed intermediate candidates before expert review. For the Manin constant conjecture, this run reproduced the proof from the statement alone, separately from the manuscript review described in our responses to R-nuya and R-dknW. In the future, we plan to pair OSVerify with larger-scale solving systems that generate multiple candidate proofs in parallel, enabling broader exploration and more effective test-time scaling through verification-guided search.

Kamnitzer's ICM 2022 conjecture. GPT-6 Pro served as the solver, and OSVerify reviewed the proofs separately. In round 1, GPT-6 Pro drafted and reviewed the proof, and Codex with GPT-6-Astra checked it again; OSVerify nevertheless found an error that invalidated the proof of the main theorem. In round 2, GPT-6 Pro revised the proof using OSVerify's feedback. Three further rounds of ChatGPT review and a Codex review all judged the revised proof correct, yet OSVerify found a second proof-breaking error and proposed a fix. After the fix was applied, a human expert verified the corrected proof. This case shows that OSVerify can catch proof-breaking errors in research mathematics that GPT-6 Pro and Codex missed.

Reference

[1] Ju et al. (2026). Automated Conjecture Resolution with Formal Verification.

I thank the author for providing those additional experiments and explanations. Most of my concerns have been solved.

Reviewer_dknWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies verification when experts cannot check every AI-generated solution. It treats valid solutions as the positive class and maximizes TNR subject to a low FNR, so invalid drafts are filtered while valid ones are kept for review. OSVERIFY splits this into hierarchical error proposal and per-error validation, and optionally trains the validator with GRPO. On Putnam100, Hard2Verify, and FirstProof-Eval it reports large cuts in invalid solutions sent to humans relative to voting. It also reports errors found in two research papers and claims help on three Langlands conjectures.

Critical strengths
  1. The paper studies a practical and important problem. With recent AI systems generating a large number of candidate proofs, solutions, and scientific hypotheses, expert verification is becoming a major bottleneck. The objective of rejecting incorrect outputs while keeping a very low false-negative rate is well motivated and meaningful.

  2. The proposed framework is intuitive and shows promising practical impact. The idea of separating error proposal and error validation is reasonable and technically sound. In addition, the reported applications to published mathematical papers and open conjectures suggest that the method may be useful in realistic expert-level settings, beyond standard benchmark evaluations.

Critical weaknesses
  1. Table 1 chooses the threshold from the valid solutions in the evaluation set, so the test labels leak into the operating point. Appendix B.4 is closer to deployment because it sets the threshold on held-out valid solutions, but it never asks how large that calibration set should be relative to the test set. It also calibrates Putnam with Putnam training data, which is too close to the test set. The paper does not show what happens if the threshold is set on other in-domain data, or on less related data. It is therefore unclear whether the reported gains would survive a more realistic calibration setup, and OSVERIFY still looks limited in practicality and generalization.
  2. OSVERIFY recursively splits the solution and sends each piece out to look for errors. Many hard errors are multi-hop: no single fragment is enough, and the mistake only appears when several parts are read together. Those are often the errors that cost experts the most time, and OSVERIFY has no real treatment of them. The paper also never converts the rejected invalid solutions into expert time saved. If the method mainly removes the easy cases, the help to experts is limited.
  3. Positive training labels on invalid solutions come from the same model family, with the reference answer in context, and an error is kept only if at least 3 of 5 samples call it substantial. That distills a reference-informed copy of the model into a reference-free copy; it is not a human label of a substantial error. Most of the training data comes from the Putnam training split, so the validator improves a lot on Putnam100, which is the same source. On other unseen datasets, the gain is small. The large boost therefore looks more like in-distribution fitting of this proxy label than a notion of a substantial error that transfers.
  4. The largest gain is on Putnam100. On Hard2Verify the gain is smaller, and the trained validator is slightly worse. Putnam100 pairs official solutions with invalid drafts from three other models, then filtered by GPT-5.5. Hard2Verify uses LLM text on both sides. The paper does not check whether OSVERIFY is partly picking up human versus model writing style rather than mathematical validity.
  5. The real-world cases take a lot of space but do not have strong controls. The paper-review study only compares against the same backbone used through its web interface. There is no comparison to a standard agent setup such as ReAct, so it is unclear whether OSVERIFY has a real advantage over that simpler alternative.
Questions for authors

Refer to the weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer dknW (1/4)

We thank the reviewer for the detailed feedback and for recognizing the motivation for screening invalid candidates while retaining valid ones for expert review. We address the concerns about calibration, multi-step errors, validator training, potential writing-style confounding, and the controls in our real-world case studies.

W1. Table 1 chooses the threshold from the valid solutions in the evaluation set, so the test labels leak into the operating point. Appendix B.4 is closer to deployment because it sets the threshold on held-out valid solutions, but it never asks how large that calibration set should be relative to the test set. It also calibrates Putnam with Putnam training data, which is too close to the test set. The paper does not show what happens if the threshold is set on other in-domain data, or on less related data. It is therefore unclear whether the reported gains would survive a more realistic calibration setup, and OSVERIFY still looks limited in practicality and generalization.

Thank you for raising this distinction. We clarify the purpose of our evaluation protocol and provide additional experiments with thresholds calibrated on held-out data.

  1. Table 1 measures attainable TNR under empirical FNR constraints, rather than deployment-calibrated performance. Following established low-error ROC evaluation practices (e.g., [1]), we report the highest attainable empirical TNR under a prescribed empirical FNR cap. This characterizes the verifier's ability to distinguish valid from invalid solutions in the low-FNR regime. Selecting a fixed threshold for future inputs is a separate calibration problem, studied extensively in prior work [2–4].

  2. Held-out calibration and calibration-set size. Appendix B.4 already applies the order-statistic calibration of FactTest [4] using 100 valid Putnam solutions whose problems are disjoint from Putnam100. At a target FNR of 5%, the resulting test (FNR, TNR) is (0.0%, 0.0%) for Voting and (4.0%, 77.6%) for OSVerify. The required calibration size depends on the target FNR and confidence level, not on the size of the test set: assuming calibration and test solutions are exchangeable, obtaining a nontrivial calibrated threshold with 90% confidence requires at least 45 valid calibration examples for a 5% FNR target, or 22 for a 10% target. The same exchangeability assumption is why calibration data should come from the same domain as the test data; thresholds calibrated on less related data are not expected to control FNR for any verifier.

  3. Held-out calibration on H2V and FirstProof-Eval. To evaluate calibration beyond Putnam, we use 24 valid AOBench proofs to calibrate H2V and 14 valid proofs from First Proof Batch 3 and the Community Experiment to calibrate FirstProof-Eval, with all calibration problems disjoint from their test sets. Since these sets are smaller than required for the finite-sample guarantee above, we use empirical calibration: we select the least restrictive threshold that meets the target empirical FNR on the calibration set, then apply this fixed threshold to 2,000 test replicates, each subsampling 16 recorded judgments per proof (Voting) or per proposed error (OSVerify).

    Test benchmarkTarget FNRVoting test FNR / TNROSVerify test FNR / TNR
    H2V5%4.0 ± 1.7 / 71.2 ± 1.50.0 ± 0.0 / 83.3 ± 0.5
    H2V10%7.7 ± 2.4 / 78.3 ± 1.50.0 ± 0.0 / 84.1 ± 0.5
    FirstProof-Eval5%3.9 ± 1.7 / 36.6 ± 1.11.2 ± 1.6 / 55.9 ± 1.7
    FirstProof-Eval10%6.3 ± 1.2 / 45.0 ± 1.72.8 ± 2.0 / 58.3 ± 1.4

    Across both benchmarks and FNR targets, OSVerify achieves a higher mean test TNR and a lower mean test FNR than Voting. Compared with thresholds selected on the evaluation set (Table 1), held-out calibration lowers OSVerify's TNR by only 0.6–2.5 pp, indicating that the reported gains persist under a more realistic calibration setup.

References

[1] Carlini et al. (2022). Membership Inference Attacks from First Principles.

[2] Tong et al. (2018). Neyman–Pearson Classification Algorithms and NP Receiver Operating Characteristics.

[3] Angelopoulos et al. (2024). Conformal Risk Control.

[4] Nie et al. (2025). FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees.

AuthorsOpenReview ↗

Response to Reviewer dknW (2/4)

W2. Many hard errors are multi-hop. Those are often the errors that cost experts the most time, and OSVERIFY has no real treatment of them. If the method mainly removes the easy cases, the help to experts is limited.

Thank you for raising the concern about errors that involve dependencies across multiple steps. We clarify three points.

  • OSVerify can detect errors involving multi-step dependencies. OSVerify does not verify decomposed fragments in isolation. At every node, the proposer and validator retain the full solution as context, while the selected segment specifies where to focus (Section 3.1). This allows OSVerify to reason about dependencies across different parts of a proof.

    For instance, on FirstProof-Eval, OSVerify identified a false connectivity bound by checking a claim about geometric fixed points against an earlier dimension requirement. In our research-paper review (Appendix E.1, Example 1), it identified an inconsistency spanning an auxiliary lemma, an intermediate diagram, and a subsequent Koszul sign correction, which the original author confirmed. These examples show that OSVerify can identify errors that require reasoning across multiple steps, rather than only isolated local mistakes.

  • OSVerify identifies subtle errors in math research missed by repeated frontier-model reviews. In our real-world case studies, OSVerify identified substantial errors that repeated frontier-model reviews failed to detect, indicating that it is not limited to filtering easy errors. For example, in the Manin constant conjecture case, 12 independent checks using Codex with GPT-6-Astra and Claude Opus 5.5 failed to identify an incorrect rationality claim, whereas OSVerify using GPT-5.6-Sol identified the error and proposed a repair. This result has been submitted to a top-tier mathematical journal.

  • Errors that are easy to verify can still be costly to discover. An error that is straightforward to check once identified may still be difficult and time-consuming to locate. Research proofs often span tens or even hundreds of pages, and experts must trace extensive arguments to find even a simple local mistake. By identifying such errors before human review, OSVerify can substantially reduce the search effort required from experts.

AuthorsOpenReview ↗

Response to Reviewer dknW (3/4)

W3. Positive training labels on invalid solutions come from the same model family. That distills a reference-informed copy of the model into a reference-free copy; it is not a human label of a substantial error. Most of the training data comes from the Putnam training split, so the validator improves a lot on Putnam100, which is the same source. On other unseen datasets, the gain is small.

Thank you for raising this concern. We would like to clarify the motivation behind our training design and the evidence for generalization beyond the training distribution.

  1. Reference-assisted supervision without human annotations or stronger teacher models is an intentional design choice. Our goal is precisely to improve a validator using only the base model and available reference solutions, without relying on stronger teacher models or additional human error-level annotations. We use the same base model, with reference solutions available during training-data construction, to generate and screen proposed errors. We then post-train the validator via RL on reference-free inputs, enabling it to assess proposed errors on unseen problems without access to reference solutions. This setting is particularly relevant to frontier scientific problems, where stronger teacher models may be unavailable and obtaining expert annotations for individual errors is prohibitively expensive. While the automatically constructed labels may contain noise, our objective is to learn useful error-validation capabilities under these limited supervision conditions.

  2. Strong in-distribution gains demonstrate the effectiveness of error-level training. The substantial improvement on Putnam100 demonstrates the effectiveness of our error-level training procedure. Although Putnam accounts for the majority of training examples, it contributes only 34.4% of the positive (substantial-error) examples, so substantial-error supervision is not restricted to Putnam. We evaluate generalization separately below.

  3. Validator training generalizes to held-out research-level mathematics. All training sources consist of competition mathematics, making research-level proofs such as FirstProof a substantially different evaluation domain. On FirstProof-Eval, validator training improves TNR from 50.4% to 57.8% at zero empirical FNR (+7.4 pp) and from 58.0% to 62.3% under a 5% empirical FNR cap (+4.3 pp) (Table 1). Since the proposer is held fixed throughout training, the attainable gains from validator training are bounded by the proposer's ability to identify substantial errors. In addition, our training-data audit ablation (Appendix B.5) shows that reference-assisted error screening improves the trained validator's TNR on FirstProof-Eval from 60.0% to 62.3% relative to training without screening, providing further evidence that reference-assisted supervision benefits verification beyond the training distribution.

Together, these results show that reference-assisted error-level training improves verification both in distribution and on independently labeled, held-out research-level problems, without human error annotations or stronger teacher models.

Reviewer_nuyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses hierarchical decomposition for more efficient answer verification, which shows better performance against voting. However, the idea is not that novel, and more experiments are needed to justify the usefulness of this method for really advanced LLMs (when faced with even more frontier questions).

Critical strengths

Generally, from the results, the proposed method works better, and some case studies are given to demonstrate its effectiveness.

Critical weaknesses

I have the following concerns.

  • I think the novelty is limited. Some core ideas (hierarchical decomposition, reviewing proposed errors) have shown in prior works, including DeepSeek Math v2 and other auto-proof agents/pipelines. I am not sure whether this is something new.
  • The practical value over stronger judge models (e.g., GPT 6 Astra, GPT 6.1 sol) remains unclear. I understand that the use of weak judges is to simulate a scenario where the task difficulty is beyond the model capacity. However, this does not mean that the conclusion can generalize to stronger judge models.
  • The calculation costs are not matched between baselines and the proposed method, especially when you use weak judges.
  • For paper reviewing, the baseline is a simple two-round web-interface review, without a matched compute budget. For conjecture solving, there is no comparison with the same solver without OSVerify or with an alternative verifier.
Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer nuya (1/2)

We thank the reviewer for the feedback and for recognizing the improvements on mathematical benchmarks and the value of our real-world case studies. We would like to clarify the novelty of OSVerify and its practical value with frontier models, and provide additional experimental results.

W1. I think the novelty is limited. Some core ideas (hierarchical decomposition, reviewing proposed errors) have shown in prior works

Thank you for pointing out these precedents. Our contribution is to formulate open-ended solution verification as a one-sided problem under limited expert review, and to develop a framework that decouples error proposal from error validation, which enables independent scaling of both components and error-level post-training to improve the TNR–FNR trade-off. Specifically:

  1. A one-sided objective motivated by limited human review. Unlike conventional verification, which primarily optimizes accuracy, our setting accounts for the asymmetric costs of verification errors in scientific research: rejecting a valid result may discard an important discovery, whereas passing an invalid one incurs additional review cost. We therefore adopt a Neyman–Pearson-style objective that maximizes TNR subject to a low FNR constraint (Section 2). Equation (1) links TNR and FNR to the expected number of candidates experts must review per retained valid solution, a key quantity when valid solutions are rare and expert review is expensive.

  2. We exploit a generation–verification gap within verification itself. Finding a substantial error requires searching across the whole solution, whereas validating a specific error hypothesis is a more focused task. OSVerify therefore decouples verification into high-recall error proposal, which searches broadly via model-guided hierarchical decomposition, and high-precision error validation, which separates substantial errors from minor or hallucinated ones (Section 3). Repeated per-error judgments are aggregated into a rejection score for conservative screening, and each component can be scaled independently with additional compute. This distinguishes OSVerify from prior work:

    • DeepSeekMath-V2 [1] scores the full proof directly and uses a meta-verifier to check whether the identified issues are real and justify the score. OSVerify instead separates hierarchical error search from repeated validation of individual error hypotheses.
    • Huang & Yang’s IMO 2025 pipeline [2] generates a bug report from the full solution and optionally checks the reported issues before iterative refinement. OSVerify instead searches for errors at multiple levels of granularity and uses repeated per-error validation for conservative rejection under a low-FNR constraint.
    • Progressive Pessimistic Verification [3] recursively bisects the solution text into smaller chunks and rejects upon the first reported error. In contrast, OSVerify uses model-guided decomposition with boundaries chosen by the model, collects proposed errors across all levels and independently validates each hypothesis before deciding whether to reject the solution.

    Moreover, OSVerify can be applied as a general framework on top of existing verifiers. Figure 4c shows that augmenting Multiagent Debate and PFBV with OSVerify consistently improves their TNR under the same empirical FNR constraints.

  3. We train the error validator on individual proposed errors. As described in Section 3.2, we use reinforcement learning to teach the validator to distinguish errors that invalidate a solution from minor issues and spurious objections. We construct error-level supervision through reference-assisted screening without requiring human annotations for each proposed error. As shown in Table 1, with the proposer held fixed, training the gpt-oss-120b validator raised TNR on the held-out research-level FirstProof-Eval benchmark from 50.4% to 57.8% at zero empirical FNR and from 58.0% to 62.3% under a 5% empirical FNR cap.

References

[1] Shao et al. (2025). DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning.

[2] Huang and Yang (2025). Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline.

[3] Huang et al. (2025). Pessimistic Verification for Open-Ended Math Questions.

AuthorsOpenReview ↗

Response to Reviewer nuya (2/2)

W3. The calculation costs are not matched between baselines and the proposed method.

Thank you for raising the question of inference cost. The submitted paper already includes compute-matched comparisons. Figure 4b and Appendix B.2 compare Voting and OSVerify under matched estimated token budgets. On Putnam100, at approximately 128k tokens per solution and an empirical FNR of at most 5%, Voting achieves 41.4% TNR, compared with 86.2% for OSVerify and 94.8% for OSVerify with the trained validator.

W2. The practical value over stronger judge models (e.g., GPT 6 Astra, GPT 6.1 sol) remains unclear.

W4. paper reviewing without a matched compute budget. For conjecture solving, there is no comparison with the same solver without OSVerify or with an alternative verifier.

We appreciate the reviewer's concerns about whether OSVerify remains useful with stronger judges and whether the real-world case studies include sufficient comparisons.

  1. Why we use weaker judges in controlled benchmarks. In our controlled benchmarks, we pair weaker judges with challenging math problems that have known solutions, so that the task exceeds the judge's reliable problem-solving ability while ground-truth labels remain available. This setting is analogous to a strong judge verifying specialized problems from areas underrepresented in its training data. The design follows established precedents in weak-to-strong generalization [4] and scalable oversight [5], where weaker models serve as proxies for supervisors facing tasks beyond their capabilities. To evaluate frontier judges directly, we complement these experiments with real-world mathematical research applications using frontier-model backbones, together with the additional comparisons below.

  2. OSVerify adds value beyond frontier-model reviewers.

    To address the concern about insufficiently competitive baselines, we conducted additional ReAct and Codex experiments for mathematical paper reviewing, and provide direct comparisons with alternative frontier-model verifiers on the same candidate proofs for conjecture solving.

    Paper review. On both case-study papers, ReAct and Codex use the same backbone and reasoning effort as OSVerify. We report single-round reviews and additional rounds under matched cumulative inference-token budgets; the table reports cumulative counts of distinct issues.

    MethodZhuLiu–Wang
    ReAct (1 round)75
    ReAct (compute-matched)1513
    Codex (1 round)1014
    Codex (compute-matched)3348
    OSVerify4588

    All OSVerify findings were confirmed by human experts and the original authors. Due to the limited rebuttal period, the ReAct and Codex candidates have not been verified by human experts, so their counts are upper bounds on their confirmed findings. Even under matched inference-token budgets, OSVerify identified more expert-confirmed issues than the total number of candidate issues reported by either baseline.

    Conjecture solving. To directly address the lack of alternative-verifier comparisons, we compare OSVerify against frontier-model verifiers on the same solver-generated candidate proofs. Alternative frontier-model verifiers repeatedly failed to detect substantial errors that OSVerify identified:

    • Manin constant conjecture. Without OSVerify, 12 independent checks with Codex (GPT-6-Astra) and Claude Opus 5.5 passed the candidate solution. However, OSVerify using GPT-5.6-Sol still found an incorrect rationality claim and proposed a repair, demonstrating its additional verification capability even compared with stronger frontier-model judges. The result has been submitted to a top-tier mathematical journal.

    • Kamnitzer’s ICM 2022 conjecture. In round 1, GPT-6 Pro drafted and reviewed a candidate proof, and Codex with GPT-6-Astra reviewed it again without finding further issues; OSVerify using GPT-6-Astra still found a major error that invalidated the proof of the main theorem. In round 2, GPT-6 Pro revised the proof based on OSVerify's feedback. Three further rounds of ChatGPT review and a Codex review found no errors, yet OSVerify found a second error that blocked the revised proof and suggested corrections.

    These comparisons show that OSVerify provides verification signals beyond repeated frontier-model reviews. Without OSVerify, the same solver would forward such flawed candidates directly to human experts, wasting expert review time.

References

[4] Burns et al. (2024). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision.

[5] Kenton et al. (2024). On scalable oversight with weak LLMs judging strong LLMs.

Dear Reviewer nuya,

Thank you again for your time and feedback. As the discussion period is approaching its end, we would greatly appreciate hearing whether our responses have addressed your main concerns, or if there are any remaining questions we could clarify.

Thank you very much for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.