acceptodds
Under review as a conference paper at ICLR 2027

RubricVerifier-STEM: Auditable Criterion-Level Rewards for STEM Reasoning

Abstract

Rubric-based rewards offer a promising way to provide dense supervision for longform STEM reasoning, but their effectiveness ultimately depends on the verifier that converts rubric criteria into reward. We study rubric verifiers as criterionvector reward interfaces and evaluate three properties required for their use in reinforcement learning: criterion-verdict correctness, surface-form invariance, and servable output compliance. To study these requirements, we introduce RUBRICHARD-STEM, a 500-example diagnostic benchmark spanning five STEM domains and 4,149 criterion labels, and develop RUBRICVERIFIER-STEM, a compact 4B verifier trained with diverse-source SFT, agreement-first RSFT, and interfaceaware RL. Despite its scale, RUBRICVERIFIER-STEM reaches 84.79% criterion accuracy and 83.94% per-example macro accuracy, outperforming all evaluated specialized verifier baselines, while using only 27% as many completion tokens as GPT-OSS-120B. Using RUBRICVERIFIER-STEM as the reward model raises Qwen3-4B-Thinking’s average score across 17 STEM benchmarks from 38.26% to 47.52%. The resulting score is 0.42 percentage points below the GPT-OSS-120B reward run. Together, these results show that a compact, auditable criterion-level verifier can provide an effective reward interface for STEM reasoning, connecting local grading fidelity to downstream RL performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reward modeling, reinforcement learning, LLM verifiers, rubric-based evaluation, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_Lw4FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an expert-annotated STEM criterion-verification benchmark and a compact 4B verifier trained through supervised fine-tuning, response selection, and reinforcement learning. It evaluates criterion prediction, reward-serving behavior, and downstream policy learning.

Critical strengths

The benchmark contains 4,149 criterion labels with blinded expert annotation and Fleiss κ = 0.86 (§2.5). The verifier achieves 84.79% criterion accuracy using approximately 27% of GPT-OSS-120B’s completion tokens (Table 2). Evaluation extends beyond the constructed benchmark. Table 7 reports external-dataset results, while Table 6 and Appendix F.2 report a 7.32-point ranking improvement over RSFT on responses from held-out generators, with a 95% confidence interval of [3.93, 9.58]. The matched-supervision-token continuation control favors RL over additional supervised training on accuracy, verbosity sensitivity, and serving parse failures (Table 17). This provides evidence for the training objective beyond simply performing more training.

Critical weaknesses

The downstream comparison does not establish statistical equivalence. Avg17 scores of 47.52 versus 47.94 are close point estimates, but replicated training uncertainty is not reported. Appendix D explicitly describes single training runs for the separate science experiment. Cross-benchmark differences in Table 15 do not quantify training variability.

The specialist-verifier comparison partly measures interface compatibility. All benchmark models use the shared criterion-vector prompt and parser, with invalid outputs receiving zero correct criteria (Appendix B). This appropriately tests readiness for the proposed interface, but may conflate grading quality with output-format compatibility. A criterion-wise adaptation of scalar/binary rubric evaluators, reporting both accuracy and total serving cost, would clarify this distinction.

Disagreement-based mining makes Rubric-Hard-STEM a targeted stress test rather than a representative sample of routine grading. The external evaluations partially address generalization, but benchmark representativeness remains uncertain. Given existing multidimensional rubric evaluation, rubric-conditioned reward models, and rubric benchmarks [1–3], the strongest originality lies in the particular STEM resource and its operational validation.

[1] Hashemi et al. LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. ACL, 2024.

[2] R3: Robust Rubric-Agnostic Reward Models. 2025.

[3] Zhang et al. RubricBench: Aligning Model-Generated Rubrics with Human Standards. 2026.

Questions for authors

Can the downstream comparisons include uncertainty across independently trained policies? How do criterion-wise adaptations of the specialist verifiers compare in accuracy and total serving cost? Can the exact scalar-control prompts clarify the distinction between GPT-OSS’s outcome reward and the proposed verifier’s output-interface control?

Reviewer_kCGBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a three-criterion framework to evaluate the adequacy of rubric verifiers for deriving dense RL post-training rewards. They publish RUBRIC-HARD-STEM as a diagnostic benchmark, and RUBRICVERIFIER-STEM, a compact 4B verifier.

Critical strengths

The paper's contribution of investigating the failure cases of rubric-based verifiers to RL post-training is timely and useful to the community. The empirical contributions and artifacts are likewise of interest to the community.

Critical weaknesses

I did not find any critical weakness.

Minor: RcRL [Motwani et al, 2026] is already highlighting weaknesses of rubric-based RL post-training; a discussion in the related work section would be adequate. However, I do not see this as impacting the novelty of the authors' contributions.

Questions for authors
  1. What role would internal consistency (such as e.g. Bradley-Terry property) play on top of the authors' three rubric criteria?
  2. Do your reported verifier improvements likely generalise beyond the particular benchmark construction you choose?
  3. Are your reported results relative to OSS-120B statistically significant?
  4. Do what degree are reward hacking pathologies considered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.