acceptodds
Under review as a conference paper at ICLR 2027

RAWR: Reward Assignment Without Rollouts in Verifiable Domains

Abstract

Understanding and evaluating multi-step reasoning in Large Language Models (LLMs) at the level of individual steps remains a key challenge. Process Reward Models (PRMs) provide a solution by scoring each step, enabling fine-grained supervision and improved reliability. However, training them requires costly human annotation or computationally intensive rollout-based labeling. To solve this, we introduce Reward Assignment Without Rollouts (RAWR), a scalable method for automatically labeling the quality of individual reasoning steps in any verifiable domain. Its step score, net information gain (NetIG), improves upon single-reference Information Gain (IG) by comparing the most-supported correct answer against the most-supported incorrect one, yielding a robust signal even for long and structured outputs like code and SQL, where IG fails. We show that the signal produced by RAWR correlates with human judgments of step quality, and we apply RAWR labels to train PRMs that achieve the best average best-of- accuracy across eight benchmarks spanning mathematics, code generation, text-to-SQL, and scientific QA. Crucially, RAWR generates no rollouts, cutting labeling complexity to and making it up to cheaper than rollout-based methods at comparable label quality, which makes large-scale process supervision practical.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Process Reward Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wEBVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAWR for generating step-level supervision in verifiable domains without rollouts. For each question, RAWR samples multiple CoT and uses validator to judge the correctness of the final output. For every reasoning prefix, RAWR computes the score between the most supported correct answer and the most supported wrong answer. And the final score will be thresholded into binary step labels and be used to train the PRM. Comprared with manual labelling and rollout-based labelling, RAWR achieves comparable results regarding best-of-K with lower costs.

Critical strengths
  1. Clear motivation and practical method design. For domains with applicable final-answer validator, the proposed method is effective.
  2. The experiments are thorogh and ablation analysis is quite detail.
  3. The writting is overall clear and easy to follow.
Critical weaknesses
  1. The 0-1 label definition is derived from threshold, which might leads to model overconfidence. These scores might be better suitable for ranking of the trace. Is there any ablation study about this alternatives like soft labels?
  2. There might be differences in trace steps. The granularity of the "STEP" definition in different traces might be different. The author might provide more evidences showing the method robustness given different step length/step budget etc.
Questions for authors
  1. The likelihood score is sum of all tokens, which might be sensitive to the trace length. It would be interesting to see how the answer length bias would affects the model performance.
  2. How to define threshold for new problem domains? How sensitive is the method to this parameter?
Reviewer_Hx1BOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes an efficient information-theoretic approach to reward assignment in reasoning chains, based on outcome labelling in domains where formal verification of the outcome is possible (such as coding tasks). Importantly the method does not require rollouts, resulting in a reduction in human labelling cost and also computational overhead, making it cheaper and more scalable than standard process reward methods.

Critical strengths
  • efficient, lightweight method for reward assignment
  • extensive experimentation and analysis
Critical weaknesses

As interesting as the paper is, it misses critical references that are very relevant and have similar advantages in terms of efficiency and no need for human labelling, namely model confidence/certainty:

Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence. Preprint, arXiv:2508.15260.

Ni, Jingwei, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov and Mrinmaya Sachan (2026) Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models, In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), San Diego, USA, pp. 11667—11689.

While there is potentially complementarity between the two styles of approach, many of the claims of novelty in the paper (such as efficiency and no need for human labels) equally apply to these competitor approaches, and without a proper theoretical and empricial comparison of the two, the true contribution/novelty of RAWR is not clear (and additionally, the confidence-based approaches are more general-purpose than RAWR in that they don't require verifiability of the outcome).

update based on author response: I concede that my original review of the paper was perhaps overly harsh. Based on the contextualisation of this past work wrt the proposed method and the updated results, I have increased my score.

Questions for authors
  • how does the proposed method compare to confidence-based methods as above?
Reviewer_cUt3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a way to label individual reasoning steps without repeatedly generating continuations from intermediate steps. RAWR scores an existing solution by comparing how strongly the model supports sampled correct answers with how strongly it supports sampled incorrect answers. The method has a clear motivation and performs better than single-reference information gain (IG) on tasks such as code and SQL, where correct answers can have many forms. The resulting labels are used to train process reward models.

Critical strengths
  1. The problem and method fit together well. Section 3 explains why IG can misjudge a solution when there are many valid answer forms; Section 4 addresses that issue, and Figure 2 shows that IG degrades as the number of correct forms grows.
  2. Table 2 provides a useful controlled comparison: RAWR 8B and IG 8B use the same backbone and training data and mainly differ in their labels. RAWR reaches 62.3% average answer-selection accuracy, compared with 60.3% for IG. The paper also reports a clear limitation: on ProcessBench, RAWR is much weaker at identifying the exact first incorrect step than at deciding whether the whole solution contains an error, and Appendix E examines this gap.
Critical weaknesses
  1. Equation (6) takes the highest score from both a correct-answer set and an incorrect-answer set, so both sets must be nonempty. Section 5.1 says problems for which all 32 sampled answers are correct are removed, but does not clearly say what happens when all 32 are wrong. This case appears common: Appendix B, Table 7 reports that only 36.54% of SynSQL training problems have at least one correct answer among the samples. Please clarify whether all-wrong problems are removed, supplemented with a reference answer, or handled by another rule, and report how many problems remain in each dataset. Without this information, the scoring procedure, data size, and labeling cost are difficult to reproduce.
  2. The paper has not shown a stable gain over a simpler model trained only on final-answer correctness for its main answer-selection task. The authors' ORM 8B reaches 62.1% in Table 2, compared with 62.3% for RAWR 8B; the paper's statistical test does not establish a difference. This does not erase RAWR's gain over IG, but it leaves unclear what the step labels add beyond final-answer labels in this setting.
Questions for authors
  1. When all sampled answers to a problem are incorrect, how does RAWR produce step labels, and how many such problems occur in each training dataset?
  2. In what controlled task that genuinely needs step-by-step judgments does RAWR provide a stable benefit over the final-answer model?
  3. Equation (2) scores the full answer string. Could differences in code or SQL answer length affect which answer receives the highest score, and has this been checked?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.