acceptodds
Under review as a conference paper at ICLR 2027

SciVerge: Objective Evaluation of Scientific Reasoning in Large Language Models

Abstract

Language models are beginning to contribute to scientific work, but their scientific reasoning is rated by human experts or language models rather than measured objectively. Measuring scientific reasoning often requires experimentation to determine whether the ideas a model considered have potential. Mathematics provides an alternative: when a scientific system is described mathematically, as a set of states and the actions that change them, whether an idea would work can be derived from the underlying mathematics. We present SciVerge, a benchmark that describes scientific systems mathematically to construct questions that objectively measure a language model's divergent thinking, the production of many possible approaches to a question, and convergent thinking, the narrowing of those approaches to the best one, the two components of creative thought identified in the psychology of creativity, which proposing and selecting a scientific approach require. Each question is posed in three phases: the model proposes approaches, commits to a subset of them, and attempts the committed approaches until one produces a proof or all have failed. SciVerge mathematically determines whether the approaches that are proposed and selected by a model can succeed, and decomposes divergent and convergent thinking into six metrics to evaluate the model's performance. SciVerge contains 500 evaluation instances drawn from over 100,000 generated across 21 systems in quantum computing, molecular biology, robotics, chemistry, crystallography, and number theory. We additionally construct a generation and screening pipeline that verifies the quality of each instance by proof before admission into the benchmark. Across seven models, the strongest solves 55 percent of instances. Scoring each phase reveals three failure modes, one per phase, that the solve rate alone would not, each calling for a different adjustment. Enabling reasoning improves selection and execution far more than proposal, and every model is far better at disproving a claim than at proving one: the strongest disproves 86 percent of the false claims but proves only 24 percent of the true ones.

Primary area:
datasets and benchmarks
Keywords:
AI for Science, Scientific Reasoning, Benchmark, Large Language Models, Group Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LNhCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SciVerge, a benchmark for objectively evaluating the scientific reasoning capabilities of large language models through mathematical formalization and verifiable proofs. SciVerge formulates problems drawn from multiple scientific domains as group-theoretic problems and introduces a three-stage protocol consisting of Diverge, Converge, and Execute, together with six metrics to assess a model’s ability to generate, select, and execute candidate solution approaches. The benchmark comprises over 100,000 verified instances spanning 21 group-presentation families, from which 500 instances are selected for evaluation. Experiments on seven open-weight models show that reasoning primarily improves approach selection and execution, while models perform substantially better at refuting false claims than at proving true ones.

Critical strengths
  1. Scientific reasoning and mathematical proof are both active areas of current research, and this paper combines them effectively within a verifiable Diverge–Converge–Execute evaluation framework, enabling stage-specific reasoning weaknesses and distinct failure modes of current models to be systematically exposed.

  2. The benchmark provides a relatively complete pipeline for instance generation, screening, and verification, with broad disciplinary coverage and strong reproducibility of the experimental setup.

Critical weaknesses
  1. The experiments evaluate only seven open-weight models, some of which are relatively out-of-date such as GLM 4.7, while formal mathematical reasoning is also a major focus of leading closed models, so the evaluation would be substantially more convincing with current GPT, Claude, and Gemini models and with multiple runs rather than a single trial per instance.

  2. The Diverge phase already provides the model with a predefined set of approach kinds, and several core metrics are defined over these fixed categories, making the task closer to reasoning within a predefined strategy space than to genuinely open ended divergent scientific thinking.

Questions for authors
  1. The current protocol allows at most five proposals and at most three committed approaches, but the motivation for these particular limits is unclear, so could the authors justify these choices and provide an ablation study to show that the main conclusions are robust to different settings?

  2. The Introduction and Related Work broadly characterize existing scientific reasoning benchmarks as relying on human or LLM ratings, although recent work such as FEM Bench [1] and SciCodePile [2] has also explored objective or executable verification, so could the authors add a concise comparison table to clarify how SciVerge differs from these methods and where its main novelty lies?

  3. Some reasoning enabled runs exhibit high parsing failure and token truncation rates, so could the authors further analyze whether these failures primarily reflect limitations in model reasoning or artifacts introduced by output formatting and token budget constraints?

Citation

[1] Mohammadzadeh, S., Hamdi, E., Shor, J., & Lejeune, E. FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs. 2025.

[2] Sun, W., et al. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation. 2026.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LNhC

We thank reviewer LNhC for recognizing the verifiable evaluation framework and broad disciplinary coverage.

W1. Closed models. We thank the reviewer for the suggestion. We evaluated two closed-source frontier models on a 100-instance subset (20 per question type, balanced true/false, 3 runs each), with two open-source models on the same instances:

ModelFluencyFlexibilityPotentialDiscriminationRestraintEstablishment
Claude Opus 5.589.0 to 92.862.4 to 66.050.5 to 52.077.4 to 78.276.5 to 79.377.0 to 83.0
GPT-6.1 Terra53.4 to 55.635.2 to 37.150.7 to 55.450.0 to 54.68.1 to 16.051.0 to 54.0
Qwen3.8-Flash-Next70.851.447.259.249.658.0
Qwen3.8-27B68.648.548.860.649.650.0

Calls that ended in an API refusal were retried up to three times, and the one instance that still refused is counted as a failure. Across three runs, every metric varies by under 8 points. For the open models, two additional runs of Qwen3.8-27B with different seeds on a 200-instance subset differ from the original run by up to 6.5 points in establishment (Appendix A.7, Table 18). Interestingly, potential is similar across all four models (47 to 55), while discrimination separates them (50 to 78). Claude Opus 5.5 selects far above chance, and GPT-6.1 Terra, with similar potential, selects at chance. Results and analysis have been added to Section 3.2 (Table 6).

W2. Predefined approach kinds. This is a fair point. Open-ended proposals can only be verified by executing each one, and a failed execution doesn't necessarily mean that the idea cannot succeed. SciVerge is a step towards verifiable evaluation of the divergent and convergent process, and this design allows us to determine whether a proposal has the potential to succeed. We clarified this framing and positioned fully open-ended divergence as future work in Section 5.

Q1. Proposal and commitment limits. The limit of five proposals is set below the number of approach kinds for most question types (5 to 9), so that a model cannot fill its plan by listing every kind, while the limit of three commitments gives the same number of certificate attempts as the closed task. Rescoring Qwen3.8-27B's original run with only its first k proposals and first m commitments:

(k, m)FlexibilityPotentialDiscriminationRestraintEstablishment
(5, 3)51.449.063.953.850.2
(5, 2)51.449.064.763.348.4
(4, 3)48.850.063.951.250.0
(3, 3)40.652.564.343.549.6
(3, 2)40.652.564.553.748.4

The rationale is now stated in Section 2.3, and the rescoring is in Appendix A.7 (Table 19).

Q2. Comparison with verifiable benchmarks. We thank the reviewer for these references.

BenchmarkVerified byCertifies negative answersScores unexecuted approachesInstances
FEM-BenchReference outputsNoNoFixed
SciCodePileUnit testsNoNoFixed
FrontierScienceLLM judgeNoNoFixed
IdeaBenchLLM judgeNoRated by judge, not verifiedFixed
DiscoveryWorldSimulator; knowledge questions graded by LLMNoNo120 from 24 templates
miniF2F / PutnamBenchProof assistantPartlyNoFixed
SciVergeCertificate checkerYesYesGenerated

The benchmarks check what the model produced or did. SciVerge additionally knows which approaches can succeed on each instance, so it can also score based on the approaches a model proposed but did not pursue. We have revised the characterization of prior work in the Introduction and Section 4 and added this comparison (Table 1).

Q3. Parsing failure and truncation. Formatting failures make up only 0.4% of all plan calls across both settings. For Qwen3.5-9B and GLM-4.7-Flash, degenerate repetition begins at a median of 5k to 7k tokens of the 64k budget, while 90% of completed plans finish under 14k tokens, so a larger budget would likely not change the plan call. In execution, truncation for the three strongest open models mostly occurs while reasoning is still in progress, while for the other models it is mostly degenerate repetition.

ModelPlan producedFormatting failureBudget reached with degenerate repetitionBudget reached other
Qwen3.8-27B97.01.20.61.2
Qwen3.8-Flash-Next97.60.21.01.2
DeepSeek-V4-Flash96.00.02.61.4
Gemma-4-31B94.00.05.80.2
Qwen3.6-27B81.61.414.82.2
GLM-4.7-Flash27.00.870.41.8
Qwen3.5-9B8.20.091.20.6

The full breakdown for plan and execution calls is added to Appendix A.6 (Tables 14 and 15).

I thank the authors for their hard work during rebuttal. I will keep my score and wish you good luck!

Reviewer_PHFBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SciVerge evaluates models’ ability to propose, select, and execute approaches to scientific questions represented through group presentations. Motivated by the difficulty of objectively evaluating research ideas, it introduces mechanically checkable certificates and a three-phase protocol with six metrics. The benchmark contains 100,012 generated instances across 21 families; seven models are evaluated on 500 balanced instances. The strongest achieves 55.2% establishment, comprising 24.4% on true claims and 86.0% on false claims. The authors report phase-specific failure modes and greater benefits from reasoning for selection and execution than for proposal generation.

Critical strengths

[Verifiable Evaluation]

  • Explicit derivations and counterexample representations make accepted answers auditable without relying on subjective expert or model judgments. Constructing certificates before generating questions, and verifying them before admission, also ensures that admitted instances have certified answers. This provides a principled pipeline for generating fresh evaluation instances across multiple question types and scientific systems.

[Informative Process Evaluation]

  • Recording proposals, commitments, and execution provides useful evidence beyond final-answer accuracy. Table 4 illustrates how a model can propose a workable approach but select an unsuccessful alternative. The appendices strengthen the reporting through budget sensitivity, closed-task comparisons, parsing and truncation statistics, and explicit disclosure of viability-label exclusions.
Critical weaknesses

[Insufficient Evidence for Phase-Specific Bottlenecks]

  • Sections 3.2 and 5 interpret metric profiles as identifying distinct bottlenecks and corresponding improvement priorities. However, discrimination measures the fraction of commitments that are viable, while establishment measures whether any commitment solves an instance. Their numerical proximity does not establish reliable execution of viable approaches. Reporting instance-level coverage of viable proposals and commitments, followed by establishment conditional on retaining a viable commitment, would better support these diagnoses.

[Insufficient Control for Protocol Effects]

  • The three-phase protocol is central to the benchmark, but its effect on solve rate is not isolated under a matched total budget. The closed-task comparison (Appendix A.6) gives direct solving one call, while the staged protocol includes planning and up to three execution calls. Recovering comparable solve rates through additional attempts therefore does not establish that staging preserves performance under equal resources. A shared total budget, counting planning and all execution attempts, would strengthen this comparison.
Questions for authors
  1. Appendix A.5 excludes unresolved searches and determines many negative viability labels from the answer and certificate type. For the remaining negative labels, what guarantees establish non-viability within the permitted parameter space? How many labels were upgraded following model-produced certificates?
  2. Beyond the 54 unresolved catalogue pairs, what fractions of each model’s actual proposals and commitments are excluded from viability-based scoring, including freely parameterized approaches? How are empty scoring denominators handled?
2 comments
AuthorsOpenReview ↗

Response to Reviewer PHFB

We thank reviewer PHFB for recognizing the principled pipeline design and analysis.

W1. Phase-specific Bottlenecks. We thank the reviewer for the insightful suggestion. We recomputed each model's scores based on the original runs (reasoning on, percent of 500 instances):

ModelViable proposalViable commitmentEstab. given viable commitmentEstab.
Qwen3.5-9B7.46.216.11.0
GLM-4.7-Flash21.815.63.80.6
Qwen3.6-27B64.858.252.230.4
Gemma-4-31B63.457.655.932.2
DeepSeek-V4-Flash81.471.456.940.6
Qwen3.8-27B86.679.063.550.2
Qwen3.8-Flash-Next85.078.070.855.2

The conditional view shows that Qwen3.5-9B and GLM-4.7-Flash fail most instances without proposing a viable approach. For the three strongest models, the limiting factor is executing a viable commitment. It also shows that Gemma-4-31B and Qwen3.6-27B lose instances at similar stages. We have added this table to Section 3.2 (Table 5), and revised the analyses in Sections 3.2 and 5.

W2. Matched total budget. We thank the reviewer for pointing this out. We agree that budget-matching provides a cleaner comparison. Complete runs, counting all completion tokens (planning, reasoning, and every attempt):

ModelTaskTokens per instanceCertified (%)
Qwen3.8-27BClosed, one call39.5k56.4
Qwen3.8-27BClosed, up to three calls81.9k65.6
Qwen3.8-27BPlan protocol94.6k56.6
DeepSeek-V4-FlashClosed, one call45.9k42.6
DeepSeek-V4-FlashClosed, up to three calls97.4k63.2
DeepSeek-V4-FlashPlan protocol110.3k47.4

At a similar budget, the closed task certifies more. This is expected, as the closed retry is able to change its approach and can adjust based on the checker's rejection output. The remaining gap reflects that committing to approaches beforehand is a more challenging task than solving freely. The model ranking is the same under both tasks at every budget from 32k tokens up. The comparison and table are added to Appendix A.7 (Table 20).

Q1. Non-viability guarantees. Of the 1,500 non-viable labels, 1,300 were proven by soundness (Appendix A.5). The remaining 200 are proven in two ways:

  • Exact abelianization check (146). Ruling out every possible count ensures no certificate exists. Every count is a homomorphism to an abelian group, and all of these factor through the abelianization, which is computed exactly from the relations by integer matrix reduction.
  • Exhaustive enumeration (54). Ruling out every world of the stated kind ensures no certificate exists. For symmetry worlds, we enumerate all homomorphisms onto each group on the fixed list (Z2, Z3, S3, S4, A4, D4). For worlds of at most 4 points, we enumerate every action.

Across the 14 runs, 411 commitments were credited as viable following model-produced certificates (2.9% of commitments). This breakdown is added to Appendix A.5 (Table 12), which now states the crediting rule.

Q2. Viability-based scoring exclusions. We thank the reviewer for the question. Below are the percent of each model's proposals and commitments excluded from viability-based scoring (reasoning on/off):

ModelProposals excluded (%)Commitments excluded (%)
Qwen3.5-9B21.6 / 15.312.1 / 13.0
Qwen3.6-27B16.7 / 10.313.2 / 8.8
Qwen3.8-27B24.4 / 24.520.6 / 20.5
Qwen3.8-Flash-Next23.8 / 18.717.8 / 13.3
DeepSeek-V4-Flash11.8 / 24.47.5 / 10.2
Gemma-4-31B20.9 / 26.620.5 / 28.1
GLM-4.7-Flash12.4 / 22.78.4 / 15.8
All runs20.315.6

54 catalogue pairs account for 1.8% of proposals and 1.7% of commitments. The remaining exclusions are all from freely parameterized approaches. When a metric's denominator is empty on an instance, the metric is undefined and left out of the metric's mean. The table and per-metric counts are added to Appendix A.5 (Table 13) and we added a description of the averaging rule to Section 2.3.

Thanks for the detailed response. The new instance-level analysis in Table 5 addresses my concern about inferring bottlenecks from aggregate metrics. The comparison in Appendix A.7 (Table 20) also clarifies the cost of staging: direct solving with retries certifies more instances for both tested models while using fewer completion tokens. I appreciate the clarification on how viability labels and exclusions are handled.

My main remaining concern is how to interpret the phase-level results. In the reviewed manuscript, Appendix A.6 (Table 9) shows that Qwen3.5-9B and GLM-4.7-Flash hit the token limit on 91.8% and 72.2% of initial calls with reasoning enabled. Given those rates, it is hard to tell how much their low viable-proposal coverage reflects difficulty generating viable ideas versus difficulty returning a parsable plan within the budget.

I see SciVerge’s distinctive contribution as checking whether proposed approaches can succeed. FrontierMath [1] provides automated verification of mathematical answers, and DiscoveryBench [2] diagnoses failures across different parts of discovery workflows drawn from published papers. SciVerge adds a different kind of diagnostic, but Sections 2.1–2.3 show that all tasks share a group-presentation formalism and that models are given the available approach types. I’m still unsure how well the diagnoses carry over to scientific tasks where models have to identify the relevant approaches themselves. This limits how strongly the results support the paper’s broader claims about scientific reasoning. My final rating is 2 (Weak rejection).

[1] Glazer et al. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI.

[2] Majumder et al. (2025). DiscoveryBench: Towards Data-Driven Discovery with Large Language Models.

Reviewer_38JVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

SciVerge evaluates models on group-theoretic problems framed as scientific tasks, separately scoring approach proposal, selection, and execution using checkable certificates.

Critical strengths
  • Separating proposal, selection, and execution offers more insight than reporting solve rate alone.
  • The generator supports fresh evaluation instances, and the reasoning-mode comparisons and budget analysis are useful.
Critical weaknesses
  • I wonder what is the definition of "scientific reasoning" and how much scientific reasoning does this work actually measure? The tasks share a group-theoretic structure, with scientific descriptions wrapped around them. Could the authors compare against purely mathematical prompts and show whether the scores predict performance on broader scientific tasks?

  • Models receive a menu of 5–9 approach kinds, while freely parameterized proposals are excluded from several metrics. I wonder could a simple fixed-menu baseline score well without generating meaningful new ideas? Have authors made any experiments to test this aspect?

  • DiscoveryWorld [1] already evaluates scientific investigation in simulation with automatic task-completion and action metrics. Could the authors position the contribution more specifically around verifying unexecuted approaches, rather than suggesting scientific reasoning is evaluated only through ratings?

  • Appendix A.6 shows that the affected models mostly hit the token limit before producing a parsable plan. Could the authors separate truncation and formatting failures from genuinely proposing too few approaches (not clear from the current results)?

Minor Comments: -Appendix A.5 uses bounded searches, including restricted matrix entries and a 10-second cap. How is “no certificate found” distinguished from “no certificate exists”? Excluding 54 unresolved pairs helps, but what guarantees the remaining negative labels? Updating labels when models find certificates also makes scores depend on which models have been evaluated.

[1] DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents, 2024

Questions for authors

some more questions beyond what in above in weaknesses:

  • I wonder if the metrics change if instances are given as bare group presentations, without the science story? any experimental results on this?

  • Also not clear what fraction of instances can a standard symbolic group solver handle?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 38JV

We thank reviewer 38JV for recognizing the insightful evaluation method and useful ablations.

W1/Q1. Scientific Reasoning Definition and Mathematical Description. By "scientific reasoning," we refer to the cognitive process used to produce scientific discoveries, which the psychology field separates into divergent and convergent thinking. SciVerge is a first step towards measuring both verifiably, on real scientific tasks (Appendix A.3) that share underlying group-theoretic structure. An ablation with Qwen3.8-27B (reasoning on) on all 500 instances with only the mathematical presentation:

FluencyFlexibilityPotentialDiscriminationRestraintEstablishment
Scientific Description71.551.449.063.953.850.2
Mathematical Description74.753.951.273.567.549.4

Establishment and the divergent metrics are similar across the two settings, so what the model can solve is a property of its reasoning rather than of the wording, which is the precondition for the scores to transfer to other scientific settings. Discrimination and restraint are higher in the mathematical setting, suggesting the scientific description may add complexity that leads the model to commit to more approaches that cannot succeed. We have clarified the definition in the Introduction and added the ablation to Appendix A.7 (Table 17).

W2. Baseline Scoring.

We would like to clarify that the metrics are designed to be analyzed holistically to provide a profile of model behavior. Potential, discrimination, and restraint each require the ability to propose and identify valid approaches. As a result, any method invariant to the task will either propose very little or commit without selecting.

BaselineFluencyFlexibilityPotentialDiscriminationRestraint
Propose 5 kinds, commit 3 at random100.072.240.340.340.1
Propose 1 kind, commit it20.014.417.417.40.0
Qwen3.8-27B, reasoning on71.551.449.063.953.8

Proposing 5 and committing at random selects at chance, as discrimination is equal to potential. Proposing the first listed kind and committing to it has low fluency and flexibility. A fixed baseline does not select above chance, and selection ability is what most separates the stronger models in our results. Both baselines are now reported in Appendix A.7.

W3. Positioning. We thank the reviewer for the suggestion and reference. We have refocused the contribution in the Introduction, revised the description of other works in the Introduction and Section 4, and added a comparison table (Table 1).

W4. Truncation and Formatting. We clarify that formatting failures are rare: at most 1.4% of plan calls for any model. Most of Qwen3.5-9B's and GLM-4.7-Flash's plan calls instead reach the token budget in degenerate repetition, where the model repeats the same text until the budget runs out (91.2% and 70.4%). When these models successfully produce plans, both propose a normal number of approaches and select near chance, so the low fluency reflects truncation rather than proposing too few approaches:

Model (reasoning on)CountFluencyFlexibilityPotentialDiscriminationRestraintEstablishment
Qwen3.5-9B4183.465.143.041.126.312.2
GLM-4.7-Flash13567.047.941.035.724.12.2

The partition for all seven models is in Appendix A.6 (Table 14).

Q2. Symbolic Group Solver. By construction, all instances have answers that are computable. On a 1,838-instance sample of the corpus, standard tools (GAP for finite quotients, coset enumeration and element order, and the Vampire prover, together with abelianization and a direct derivation search) resolve 1,650 instances (89.8%) within a fixed time budget. An instance is admitted only once its answer has been certified computationally (Section 2.4). This allows SciVerge to compute the viability of each proposed approach automatically.

Minor Comment. Viability Labels. A search that finds no certificate is inconclusive, so the pair is left unlabeled and never marked non-viable. Every non-viable label rests on a proof that no certificate of that kind exists: 1,300 pairs because the certificate type cannot witness the instance's side (a counterexample world cannot prove a true claim, and a derivation cannot refute a false one), 146 pairs because the exact abelianization does not separate the words, and 54 pairs by enumerating the entire allowed space. No model's output changes the viability table: an accepted certificate credits the committed approach only within the attempt that produced it, so a model's scores do not depend on which other models have been evaluated. We clarified the labeling process in Appendix A.5 and added this breakdown (Table 12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.