acceptodds
Withdrawn from ICLR 2027

Decision-Sufficient Early Stopping for Process Reward Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Checking an LLM-generated solution often requires a process reward model (PRM) to score every reasoning step. We study when this checking can stop early while returning exactly the decision of full verification. We formalize decision sufficiency: a prefix is sufficient when every completion allowed by the verifier contract induces the same terminal decision. By bounding the terminal scores reachable from each prefix, we obtain an exact monitor that stops only when the entire reachable range lies within one decision region. Under the stated score-range model, tight closedform envelopes support minimum, maximum, sum, product, and finite-horizon mean aggregation. The monitor requires no additional training and, with exact bounds, stops at the earliest sufficient prefix under the fixed observation order and continuation model. On ProcessBench, the finite-horizon mean monitor saves 32.52% of verifier input tokens with unchanged decisions, whereas a monotonethreshold shortcut saves none. Across the main Qwen and Llama public-benchmark evaluations, the monitors save 14–33% of verifier input. On a separate 1,000- trace PRMBench timing subset under matched incremental execution semantics, they reduce Qwen and Llama mean verification time by 26.01% and 39.13%, respectively, with identical full incremental decisions and shared-prefix scores.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, process reward models, process verification, decision sufficiency, exact early stopping, reachable-state analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TCx2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the early stopping for reasoning trace, aims at reducing the time costs for non-neccessary judgement of the follow-up steps. A prefix is decision-sufficient when every allowed completion of its unseen scores yields the same final decision. The proposed method defines this using reachable score intervals. Closed-form bounds are derived for different aggregation methods. Across multiple PRMs and benchmarks, the method saves verifier input and verification time.

Critical strengths
  1. The paper removes structural verification redundancy, giving an interesting and novel direction distinct from statistical calibration or predictive early-exit methods.
  2. The proposed method is simple, training-free, and easy to integrate with existing PRM verifiers.
Critical weaknesses
  1. The paper lacks a comparison with one-pass full verification, which may be more efficient than incremental verification in practical deployment. Moreover, the two execution modes can produce different scores and final decisions.
  2. The main results rely largely on min/max aggregation. For the sum, product, and mean settings, the paper does not report the corresponding full verifiers’ accuracy or decision distributions.
  3. Actual runtime gains are demonstrated only for min aggregation. It remains unclear whether the more general aggregation settings provide similar wall-clock benefits.
  4. In the minimum setting, early stopping appears to certify only INVALID decisions. The authors should clarify this limitation and discuss whether early VALID or ABSTAIN decisions are possible in practical settings.
Questions for authors

Please refer to the Weaknesses.

Reviewer_PwfiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes some straightforward arithmetic techniques for online aggregation step-wise process rewards, such as running min, max, and mean. By filtering on these aggregates, it is possible to discard unpromising trajectories early, saving computation.

Critical strengths
  • The goal of saving compute in long derivations is an important one to pursue.
  • The proposed technique is straightforward, easy to implement and understand.
  • Discussion of related work is fairly comprehensive (although specific details of the relation to this prior work could be more clear)
Critical weaknesses
  • Prior work (e.g., Snell et al ICLR 2025, Setlur et al ICLR 2025) makes extensive use of beam search for the purpose of saving compute in multi-step rollouts, yet there is no comparison against beam search in the submission.
  • Conceptually, the paper simply assumes that the score for a rollout should be expressible as an elementary function of the process rewards (mean, min, max, etc), but it is not clear that this is true.
  • Empirically, the paper lacks clear comparisons against alternative methods at the core task: cost-performance tradeoffs for multi-step derivations. (The experimental section of the paper is as bit difficult to follow; the key evidence seems to be Table 4 but as far as I can tell it doesn't answer this question.)
Questions for authors

How does the proposed monitoring technique compare against state-of-the-art alternatives for optimizing the compute-performance tradeoff?

Reviewer_pZw5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a way to early stop process reward models by bounding the reachable terminal scores and early terminating if a decision boundary is guaranteed. This bounding addresses several forms of aggregation such as min/max/product etc. There is a saving of tokens on ProcessBench and Qwen and Llama evaluations and also save on evaluation time.

Critical strengths
  • Are able to get gains in sampling input tokens and also wall clock time by computing a simple upper and lower bound for voting, preserving the exact selection from voting over the full sample set.
Critical weaknesses
  • The early stopping or pruning seems to have quite a weak upper/lower bound. It would be useful to compare against inexact methods and see what the cost vs performance benefit is.
  • There are very limited baselines used in the evaluation setting and a lot of methods that do adaptive sampling or search (e.g Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters).
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.