- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces LSR-Ben, a benchmark for evaluating process reward models and LLMs on scientific and logical reasoning. It contains 3,595 problem-solution pairs across four scientific and five logical subdomains, with solutions generated by 15 LLMs. Human annotators identify the first erroneous step, or judge the entire solution correct, and provide error categories and explanations. The evaluation compares PRMs and prompted LLMs, includes ProcessBench comparisons and fast/slow-thinking settings, and analyzes error-identification behavior. The reported results indicate substantial remaining difficulties in verifying reasoning processes beyond mathematics.
Critical strengths
-
The problem is important, and the dataset has potential value. Evaluating intermediate reasoning beyond mathematics is worthwhile. The coverage of nine subdomains, diverse solution generators, and human annotations of error locations, types, and explanations could provide a useful resource for studying verification failures. The use of domain-qualified annotators and multiple reviewers per solution is also a positive design choice.
-
The empirical evaluation is broad and informative. The paper evaluates multiple PRMs and general-purpose LLMs, reports results by subdomain and error category, and compares fast and slow thinking. These analyses provide useful observations about current models’ limitations, extending beyond an aggregate leaderboard.
Critical weaknesses
-
Limited differentiation from existing benchmarks. The construction pipeline and first-error metric largely follow ProcessBench. BIG-Bench Mistake [1] already includes logical first-error localization, while DeltaBench [2] evaluates scientific and general reasoning processes. Although their protocols and coverage differ, these omitted precedents weaken the claimed research gap and leave LSR-Ben’s added contribution insufficiently established.
-
Insufficiently specified domain-specific annotation criteria. The subdomains, expert annotators, and error taxonomy are useful, but the operational criteria need clarification: how are unsupported hypotheses distinguished from erroneous steps in inductive or abductive reasoning, and how is scientific knowledge verified? These distinctions are central to the validity of the benchmark’s labels.
-
Unsubstantiated solution-coverage ratings. Table 1 links coverage qualitatively to generator number and diversity without defining the mapping to one, two, or three stars. Appendix A.1’s PCA visualization shows differences between generators, but does not establish broader coverage of reasoning strategies or error patterns.
-
Insufficient support for domain-specific interpretations. The ProcessBench comparison changes domains, sources, generators, and solution lengths (7.1 versus 13.7 steps). Without controlling for length, error position, and generator, the performance gap cannot isolate domain-specific verification difficulties. Section 4.1 also misclassifies FoVer as mathematics-only, although it uses formal logic and mathematical theorem proving [3], further weakening the transfer interpretation.
-
Unclear annotation reliability and inconsistent statistics. Agreement before and after the two discussion rounds is not separately reported, and approximately 15% of samples are discarded. Table 2 reports 76.4% overall three-annotator agreement, whereas its subdomain counts and rates imply approximately 80.8%. Both the aggregation and the effect of filtering need clarification.
-
Unclear comparability of evaluation splits. Section 4.1 explicitly excludes threshold-tuning samples only when evaluating scalar PRMs, leaving it unclear whether all models share the same test subset. The repeated questions reported in Appendix A.6 also require clarification of question-level separation between development and test sets.
[1] Tyen et al. LLMs cannot find reasoning errors, but can correct them given the error location. Findings of ACL, 2024.
[2] He et al. Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?. ACL, 2025.
[3] Kamoi et al. Efficient PRM Training Data Synthesis via Formal Verification. Findings of ACL, 2026.
Questions for authors
-
Solution coverage: What precisely distinguishes one, two, and three stars in Table 1? What evidence beyond generator counts and the PCA visualization establishes broader coverage of reasoning strategies or error patterns?
-
Novelty: What can LSR-Ben measure or reveal beyond existing process-evaluation benchmarks? Which construction choices and experimental findings establish this added value, beyond increasing domain and generator diversity?
-
Domain-specific annotation: What rules determine the first error in scientific, inductive, and abductive reasoning? Please explain how annotators distinguish knowledge errors, invalid inferences, and tentative hypotheses, with representative boundary cases and evidence that these criteria are applied reliably.
-
Experimental attribution: Do the main conclusions hold after controlling for solution length, first-error position, and generator? Which results distinguish missing scientific knowledge from faulty logical inference? Please also specify the FoVer checkpoint and correct its training-domain classification.
-
Label reliability: Are the agreement rates measured before or after discussion? Please report the initial independent agreement and discard rates by domain and reason, and reconcile Table 2’s overall agreement with its subdomain statistics.
-
Evaluation protocol: Were all models evaluated on the same test subset, with no original question shared between development and test sets? Please specify the splits and scalar-PRM thresholds, and clarify whether the main comparisons remain stable under a common evaluation protocol.