acceptodds
Withdrawn from ICLR 2027

LSR-Ben: A Logical and Scientific Reasoning Benchmark for Evaluating Process Reward Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Currently, process reward models (PRMs) have exhibited remarkable potential for test-time scaling. Since large language models (LLMs) regularly generate flawed intermediate reasoning steps when tackling a broad spectrum of reasoning and decision-making tasks, PRMs are required to possess capabilities for detecting process-level errors in real-world scenarios. However, existing benchmarks primarily focus on mathematical reasoning, thereby failing to comprehensively evaluate the error detection ability of PRMs across diverse reasoning scenarios. To mitigate this gap, we introduce LSR-Ben, a process-level benchmark specifically designed for assessing PRM's performance across two primary reasoning domains (science and logic) and nine subdomains. We conduct extensive experiments on a diverse set of 22 models, encompassing both PRMs and LLMs, and derive two key findings: (1) In domains beyond mathematical reasoning, the error-detection ability of existing PRMs and LLMs is found to be markedly weaker by comparison. (2) In general, LLMs exhibit a tendency toward over-identification of errors compared to PRMs, whereas PRMs exhibit an inherent tendency to overlook errors compared to LLMs. We hope LSR-Ben can foster future researches on PRMs for more general domains, thereby enhancing the reasoning capabilities of LLMs.

Primary area:
datasets and benchmarks
Keywords:
process reward models, benchmark, logical reasoning, scientific reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g5yrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces LSR-Ben, a benchmark for evaluating process reward models and LLMs on scientific and logical reasoning. It contains 3,595 problem-solution pairs across four scientific and five logical subdomains, with solutions generated by 15 LLMs. Human annotators identify the first erroneous step, or judge the entire solution correct, and provide error categories and explanations. The evaluation compares PRMs and prompted LLMs, includes ProcessBench comparisons and fast/slow-thinking settings, and analyzes error-identification behavior. The reported results indicate substantial remaining difficulties in verifying reasoning processes beyond mathematics.

Critical strengths
  1. The problem is important, and the dataset has potential value. Evaluating intermediate reasoning beyond mathematics is worthwhile. The coverage of nine subdomains, diverse solution generators, and human annotations of error locations, types, and explanations could provide a useful resource for studying verification failures. The use of domain-qualified annotators and multiple reviewers per solution is also a positive design choice.

  2. The empirical evaluation is broad and informative. The paper evaluates multiple PRMs and general-purpose LLMs, reports results by subdomain and error category, and compares fast and slow thinking. These analyses provide useful observations about current models’ limitations, extending beyond an aggregate leaderboard.

Critical weaknesses
  1. Limited differentiation from existing benchmarks. The construction pipeline and first-error metric largely follow ProcessBench. BIG-Bench Mistake [1] already includes logical first-error localization, while DeltaBench [2] evaluates scientific and general reasoning processes. Although their protocols and coverage differ, these omitted precedents weaken the claimed research gap and leave LSR-Ben’s added contribution insufficiently established.

  2. Insufficiently specified domain-specific annotation criteria. The subdomains, expert annotators, and error taxonomy are useful, but the operational criteria need clarification: how are unsupported hypotheses distinguished from erroneous steps in inductive or abductive reasoning, and how is scientific knowledge verified? These distinctions are central to the validity of the benchmark’s labels.

  3. Unsubstantiated solution-coverage ratings. Table 1 links coverage qualitatively to generator number and diversity without defining the mapping to one, two, or three stars. Appendix A.1’s PCA visualization shows differences between generators, but does not establish broader coverage of reasoning strategies or error patterns.

  4. Insufficient support for domain-specific interpretations. The ProcessBench comparison changes domains, sources, generators, and solution lengths (7.1 versus 13.7 steps). Without controlling for length, error position, and generator, the performance gap cannot isolate domain-specific verification difficulties. Section 4.1 also misclassifies FoVer as mathematics-only, although it uses formal logic and mathematical theorem proving [3], further weakening the transfer interpretation.

  5. Unclear annotation reliability and inconsistent statistics. Agreement before and after the two discussion rounds is not separately reported, and approximately 15% of samples are discarded. Table 2 reports 76.4% overall three-annotator agreement, whereas its subdomain counts and rates imply approximately 80.8%. Both the aggregation and the effect of filtering need clarification.

  6. Unclear comparability of evaluation splits. Section 4.1 explicitly excludes threshold-tuning samples only when evaluating scalar PRMs, leaving it unclear whether all models share the same test subset. The repeated questions reported in Appendix A.6 also require clarification of question-level separation between development and test sets.

[1] Tyen et al. LLMs cannot find reasoning errors, but can correct them given the error location. Findings of ACL, 2024.

[2] He et al. Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?. ACL, 2025.

[3] Kamoi et al. Efficient PRM Training Data Synthesis via Formal Verification. Findings of ACL, 2026.

Questions for authors
  1. Solution coverage: What precisely distinguishes one, two, and three stars in Table 1? What evidence beyond generator counts and the PCA visualization establishes broader coverage of reasoning strategies or error patterns?

  2. Novelty: What can LSR-Ben measure or reveal beyond existing process-evaluation benchmarks? Which construction choices and experimental findings establish this added value, beyond increasing domain and generator diversity?

  3. Domain-specific annotation: What rules determine the first error in scientific, inductive, and abductive reasoning? Please explain how annotators distinguish knowledge errors, invalid inferences, and tentative hypotheses, with representative boundary cases and evidence that these criteria are applied reliably.

  4. Experimental attribution: Do the main conclusions hold after controlling for solution length, first-error position, and generator? Which results distinguish missing scientific knowledge from faulty logical inference? Please also specify the FoVer checkpoint and correct its training-domain classification.

  5. Label reliability: Are the agreement rates measured before or after discussion? Please report the initial independent agreement and discard rates by domain and reason, and reconcile Table 2’s overall agreement with its subdomain statistics.

  6. Evaluation protocol: Were all models evaluated on the same test subset, with no original question shared between development and test sets? Please specify the splits and scalar-PRM thresholds, and clarify whether the main comparisons remain stable under a common evaluation protocol.

Reviewer_26jCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LSR-Ben, a benchmark for evaluating whether process reward models and general-purpose LLMs can identify the earliest erroneous step in scientific and logical reasoning. It contains 3,595 problem–solution pairs across nine subdomains, with solutions generated by 15 models and human annotations of error locations, types, and explanations. The evaluation shows substantially lower performance than on the mathematical ProcessBench benchmark. The authors also report that, among the selected models, PRMs more frequently miss the first error, while LLMs more frequently flag an earlier, valid step.

Critical strengths
  1. A relevant extension of process-level evaluation. Evaluating reasoning verification beyond mathematics is important. While the underlying questions come from existing datasets, the added human annotations provide information that final-answer labels alone cannot supply.

  2. Substantial annotation effort and diverse solution sources. The benchmark uses three annotators per instance, a disagreement-resolution procedure, and solutions from multiple model families. These are useful design choices for constructing a process-evaluation resource.

  3. Informative diagnostic breakdowns. Results are reported by reasoning subdomain, error category, and performance on erroneous versus error-free solutions. These breakdowns offer more insight than a single aggregate score.

Critical weaknesses
  1. The cross-domain comparison does not sufficiently isolate domain generalization. LSR-Ben solutions average 13.7 steps, compared with 7.1 for ProcessBench, and differ in generation and segmentation procedures. Since evaluation requires exact localization of the first error, these differences may contribute to the performance gap. Length- and error-position-controlled comparisons would strengthen the claim that the decline primarily reflects weaker scientific and logical reasoning verification.

  2. Annotation reliability needs stronger evidence and clearer reporting. Agreement after annotators exchange explanations does not establish independent inter-annotator reliability. The paper should distinguish initial agreement from post-discussion consensus and provide an independent expert audit. There is also an apparent inconsistency in Table 2: weighting the nine reported full-agreement rates by their sample counts gives approximately 80.8%, whereas the reported total is 76.4%. This requires clarification.

  3. The interpretation of model-specific error tendencies is stronger than the evidence supports. Earlier versus missed error predictions depend on decision thresholds, prompting, and the distribution of error positions. Although thresholds are tuned for some PRMs, the reported comparisons do not establish that over-identification or under-identification is an inherent property of the model category. Comparisons at matched false-positive rates and threshold-sensitivity analyses would be more convincing.

  4. The comparability of evaluation sets is unclear. For some scalar-output PRMs, 40 examples per subdomain are reserved for threshold selection and excluded from evaluation. It is unclear whether all other models are evaluated on the same remaining examples. A shared held-out test set and uncertainty estimates are needed to support reliable comparisons.

  5. The practical significance of the benchmark remains insufficiently validated. The task largely extends an established first-error-localization protocol to additional domains. This can be a valuable contribution, but the paper does not demonstrate whether higher benchmark scores predict better solution selection, search guidance, or other downstream PRM applications. Such evidence would strengthen the case that the benchmark measures practically useful verification ability.

Questions for authors

same as above

Reviewer_pFU5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a benchmark designed for PRM in Logical and Scientific Reasoning domain. The motivation is that most existing PRM benchmarks focus on math reasoning, which usually can be verified, but in science domain this is not always true. The benchmark starts from some MMLU-pro questions, using Physics, Chemistry, Biology and Computer Science, and other logical question from five existing reasoning benchmarks.

The benchmark procedure incudes: 1. collecting questions from existing benchmarks. 2. Generate reasoning traces from 15 LLMs. 3. Use qwen3-32B to reorganize the reasoning. 4. 3 annotators and cross-validation. The resulting benchmarks includes 3595 examples.

Critical strengths
  • Human annotations are real contribution to the community.

  • The reasoning step re-organizing is an important step that aims to clarify the reasoning step boundaries. This makes the benchmark construction more solid.

  • The paper includes reasoning traces from many different LLMs, the diversity is in the design.

Critical weaknesses
  • The benchmark has limited novelty. There are already previous benchmark doing similar thing, e.g. Mr-Ben, finding the first error in reasoning traces for science and logical questions. Although they might not be designed for PRM, but the resulting benchmarks are similar.

  • A static benchmark design such as this paper, although including reasoning from many LLMs, still will face off-policy problem when applying to a new LLM. Same concern also happen in reasoning step reorganization model, qwen3-32B. The benchmark won't be as indicative after a few generations when there were gaps between reasoning patterns.

Questions for authors

N/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.