- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SciVerge, a benchmark for objectively evaluating the scientific reasoning capabilities of large language models through mathematical formalization and verifiable proofs. SciVerge formulates problems drawn from multiple scientific domains as group-theoretic problems and introduces a three-stage protocol consisting of Diverge, Converge, and Execute, together with six metrics to assess a model’s ability to generate, select, and execute candidate solution approaches. The benchmark comprises over 100,000 verified instances spanning 21 group-presentation families, from which 500 instances are selected for evaluation. Experiments on seven open-weight models show that reasoning primarily improves approach selection and execution, while models perform substantially better at refuting false claims than at proving true ones.
Critical strengths
-
Scientific reasoning and mathematical proof are both active areas of current research, and this paper combines them effectively within a verifiable Diverge–Converge–Execute evaluation framework, enabling stage-specific reasoning weaknesses and distinct failure modes of current models to be systematically exposed.
-
The benchmark provides a relatively complete pipeline for instance generation, screening, and verification, with broad disciplinary coverage and strong reproducibility of the experimental setup.
Critical weaknesses
-
The experiments evaluate only seven open-weight models, some of which are relatively out-of-date such as GLM 4.7, while formal mathematical reasoning is also a major focus of leading closed models, so the evaluation would be substantially more convincing with current GPT, Claude, and Gemini models and with multiple runs rather than a single trial per instance.
-
The Diverge phase already provides the model with a predefined set of approach kinds, and several core metrics are defined over these fixed categories, making the task closer to reasoning within a predefined strategy space than to genuinely open ended divergent scientific thinking.
Questions for authors
-
The current protocol allows at most five proposals and at most three committed approaches, but the motivation for these particular limits is unclear, so could the authors justify these choices and provide an ablation study to show that the main conclusions are robust to different settings?
-
The Introduction and Related Work broadly characterize existing scientific reasoning benchmarks as relying on human or LLM ratings, although recent work such as FEM Bench [1] and SciCodePile [2] has also explored objective or executable verification, so could the authors add a concise comparison table to clarify how SciVerge differs from these methods and where its main novelty lies?
-
Some reasoning enabled runs exhibit high parsing failure and token truncation rates, so could the authors further analyze whether these failures primarily reflect limitations in model reasoning or artifacts introduced by output formatting and token budget constraints?
Citation
[1] Mohammadzadeh, S., Hamdi, E., Shor, J., & Lejeune, E. FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs. 2025.
[2] Sun, W., et al. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation. 2026.
2 comments
I thank the authors for their hard work during rebuttal. I will keep my score and wish you good luck!
Response to Reviewer LNhC
We thank reviewer LNhC for recognizing the verifiable evaluation framework and broad disciplinary coverage.
W1. Closed models. We thank the reviewer for the suggestion. We evaluated two closed-source frontier models on a 100-instance subset (20 per question type, balanced true/false, 3 runs each), with two open-source models on the same instances:
Calls that ended in an API refusal were retried up to three times, and the one instance that still refused is counted as a failure. Across three runs, every metric varies by under 8 points. For the open models, two additional runs of Qwen3.8-27B with different seeds on a 200-instance subset differ from the original run by up to 6.5 points in establishment (Appendix A.7, Table 18). Interestingly, potential is similar across all four models (47 to 55), while discrimination separates them (50 to 78). Claude Opus 5.5 selects far above chance, and GPT-6.1 Terra, with similar potential, selects at chance. Results and analysis have been added to Section 3.2 (Table 6).
W2. Predefined approach kinds. This is a fair point. Open-ended proposals can only be verified by executing each one, and a failed execution doesn't necessarily mean that the idea cannot succeed. SciVerge is a step towards verifiable evaluation of the divergent and convergent process, and this design allows us to determine whether a proposal has the potential to succeed. We clarified this framing and positioned fully open-ended divergence as future work in Section 5.
Q1. Proposal and commitment limits. The limit of five proposals is set below the number of approach kinds for most question types (5 to 9), so that a model cannot fill its plan by listing every kind, while the limit of three commitments gives the same number of certificate attempts as the closed task. Rescoring Qwen3.8-27B's original run with only its first k proposals and first m commitments:
The rationale is now stated in Section 2.3, and the rescoring is in Appendix A.7 (Table 19).
Q2. Comparison with verifiable benchmarks. We thank the reviewer for these references.
The benchmarks check what the model produced or did. SciVerge additionally knows which approaches can succeed on each instance, so it can also score based on the approaches a model proposed but did not pursue. We have revised the characterization of prior work in the Introduction and Section 4 and added this comparison (Table 1).
Q3. Parsing failure and truncation. Formatting failures make up only 0.4% of all plan calls across both settings. For Qwen3.5-9B and GLM-4.7-Flash, degenerate repetition begins at a median of 5k to 7k tokens of the 64k budget, while 90% of completed plans finish under 14k tokens, so a larger budget would likely not change the plan call. In execution, truncation for the three strongest open models mostly occurs while reasoning is still in progress, while for the other models it is mostly degenerate repetition.
The full breakdown for plan and execution calls is added to Appendix A.6 (Tables 14 and 15).