acceptodds
Under review as a conference paper at ICLR 2027

Can Language Models Learn to Reject Their Own Bad Reasoning Steps?

Abstract

Verifier-guided decoding can prevent harmful reasoning steps from contaminating subsequent generation, but typically relies on an external learned verifier. Can a language model instead learn to reject its own bad reasoning steps? This requires distinguishing candidate steps sampled from the same problem and reasoning prefix. We define a prefix's recoverability as the probability that the frozen generator can complete it correctly. Diagnostic measurements show that most adjacent changes in recoverability are difficult to resolve with practical Monte Carlo budgets, even when cumulative degradation is detectable. Meanwhile, candidates sampled from a shared prefix exhibit a sparse low-recoverability tail, suggesting an opportunity for selective rejection. We introduce Self-Step Rejection (SSR), which trains a lightweight LoRA acceptance gate on the generator's own backbone while keeping the base model frozen. SSR uses confidence-qualified first-passage supervision: steps before the first resolved crossing of a root-relative recoverability barrier are labeled as accepted, the crossing step is labeled as rejected, and unresolved steps and the subsequent suffix are excluded from supervision. Training combines pointwise classification with same-prefix pairwise learning, followed by group-relative policy refinement using final-answer correctness. At inference, the gate accepts candidates or resamples from the unchanged prefix, subject to a per-position attempt cap and an episode-level rejection budget, without an external learned verifier. Across three reasoning models and five mathematical reasoning benchmarks, SSR improves macro-average accuracy over single-pass decoding by 5.4–10.1 percentage points using 1.21–1.40 as many generated tokens, achieving the highest macro-average accuracy among the evaluated step-level methods for each generator, while full-solution scaling methods require 4.47–8.27 the single-pass token cost for comparable performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Self-Verification, Verifier-Guided Decoding, Test-Time Computation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dn8COpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The key premise of this paper is to investigate if language models that have the ability to reason can also act as their own verifier in terms of identifying and discarding `bad' reasoning steps. This would then result in reducing the reliance on an external learned verifier. Towards this goal, the paper introduces self-step rejection (SSR) to commit to or reject a sampling step given a reasoning prefix. Through experiments, it is shown that SSR improves (macro-)average accuracy while not significantly increasing token usage over multiple mathematical reasoning benchmarks and three reasoning models.

Critical strengths
  1. The challenges associated with same-prefix candidate rejection - i.e., a setting where an external verifier is not used- is described in detail, and the paper further details how the proposed SSR overcomes these challenges, which are mainly related to rollout budgets and sparsity of the recoverability tail.

  2. The accuracy performance of SSR achieved across three reasoning models as shown in Figure 1 is impressive, while not incurring significantly high overhead in terms of token usage. At the same time, SSR is shown to achieve comparable accuracy to full-solution scaling baselines at a fraction of token usage.

Critical weaknesses
  1. The clarity of the explanation of the graphs in Fig. 2 can be significantly improved. The explanation in Sec. 2.2 and Appendix D.1 basically verbalize the shapes of the graphs, but fail to provide context in terms of anchoring the graphs to perhaps a specific example text/ paragraph.
Questions for authors
  1. Please see Weaknesses, above.

  2. Is there a corresponding ablation on the two other reasoning models similar to that for Qwen3-14B in Table 3? Similarly for Tables 4, 5 in the Appendix. In Table 5, are the numbers also computed over three runs like in Tables 4, 6,7?

3 comments
AuthorsOpenReview ↗

Response to Reviewer dn8C (Part 1)

We sincerely thank the reviewer for their careful review and valuable feedback.

Response to W1

We thank the reviewer and will add annotation for Figure 2 accordingly.

Figure 2(a). The problem requires xi<1x_i < 1; the reference answer is 11 because negative terms are unbounded. The trajectory gradually reinterprets the constraint. It first notes that negative xix_i are unbounded and wonders whether ∣xi∣<1\lvert x_i\rvert < 1 was intended (0–887 tokens, V^\hat V: 0.89→0.840.89\to0.84). It tests n=1,2n=1,2 and reaches contradictions (→0.75\to0.75), then proposes an "implicit constraint ∣xi∣<1\lvert x_i\rvert < 1" (→0.59\to0.59). V^\hat V rises temporarily when it re-reads the original statement (→0.72\to0.72), falls again when it concludes the problem is "mistranslated" and adopts ∣xi∣<1\lvert x_i\rvert < 1 (→0.59\to0.59), and declines to 0.19 as it derives n≥20n\geq20 under the wrong constraint; it finally answers 20. No single paragraph produces a statistically resolved drop. The degradation accumulates as the model commits step by step to a misreading, which is what root-relative first-passage labels are designed to capture. We note in the caption that this trajectory was selected as a failure case on a problem the model usually solves (root 0.89).

Figure 2(b). The shared prefix is at 887 tokens of the same trajectory, where the model starts testing small nn. The two low-value candidates (7/167/16 and 8/168/16) both state that the problem "should be ∣xi∣<1\lvert x_i\rvert < 1"; high-value candidates (15/1615/16 and 16/1616/16) instead carry out the case analysis and derive ∑∣xi∣−∣S∣=2⋅min⁡(P,N)\sum\lvert x_i\rvert-\lvert S\rvert=2\cdot\min(P,N) without changing the constraint. The low-value tail thus corresponds to candidates that commit to the misreading early.

We will revise §2.2–2.3 and the caption accordingly, and add the tables below to Appendix D.1.

Evaluations of the Figure 2(a) trajectory (Qwen3-14B, OlymMATH-HARD-2: xi<1x_i<1, ∑i∣xi∣=19+∣∑ixi∣\sum_i\lvert x_i\rvert = 19 + \lvert\sum_i x_i\rvert, minimum nn; reference answer 11). V^\widehat V uses 64 continuations at the root and 32 elsewhere; intervals are 95% normal approximations. Dt=V^0−V^tD_t=\widehat V_0-\widehat V_t. The barrier (1−τ)V^0=0.623(1-\tau)\widehat V_0=0.623 is first crossed at 1,735 tokens. Each row summarizes the text between the previous boundary and this one.

#TokensV^\widehat V95% CIDtD_tText since previous boundary
000.891[0.81, 0.97]0.000(root)
12090.812[0.68, 0.95]0.078Restates the problem; notes xi<1x_i<1 allows negatives
23520.875[0.76, 0.99]0.016Notes the gap of 19 implies cancellation
38870.844[0.72, 0.97]0.047Negatives are unbounded; wonders if ∣xi∣<1\lvert x_i\rvert<1 was meant
41,5860.750[0.60, 0.90]0.141Tests n=1,2n=1,2; n=2n=2 needs some xi≥9.5x_i\geq 9.5
51,7350.594[0.42, 0.76]0.297Proposes an “implicit constraint ∣xi∣<1\lvert x_i\rvert<1” (first crossing)
61,8060.719[0.56, 0.87]0.172Re-reads the original statement
71,9570.594[0.42, 0.76]0.297Calls it a mistranslation; adopts ∣xi∣<1\lvert x_i\rvert<1
82,1180.562[0.39, 0.73]0.328Under ∣xi∣<1\lvert x_i\rvert<1: ∑∣xi∣<n\sum\lvert x_i\rvert<n, so n≥20n\geq 20
92,2590.594[0.42, 0.76]0.297Sets S=∑xiS=\sum x_i; seeks a formula for ∑∣xi∣−∣S∣\sum\lvert x_i\rvert-\lvert S\rvert
102,7440.500[0.33, 0.67]0.391Splits into positive and negative parts
113,1240.344[0.18, 0.51]0.547Checks “twice the smaller side” on examples
123,2890.344[0.18, 0.51]0.547Cases on the sign of SS; one side must sum to 9.5
133,3910.281[0.13, 0.44]0.609Defines A,BA,B; writes A+B=19+∣B−A∣A+B=19+\lvert B-A\rvert
143,9360.188[0.05, 0.32]0.703Solves A=9.5A=9.5; with ∣xi∣<1\lvert x_i\rvert<1 each side needs ≥10\geq 10 terms

Candidates from the Figure 2(b) prefix (887 tokens into the trajectory in the preceding table), 16 continuations each.

Cand.V^\widehat VKey move
157/16“Wait, perhaps the problem meant that each ∣xi∣<1\lvert x_i\rvert<1?” Rewrites the constraint immediately.
288/16After n=2n=2 fails: “there must be a typo, and the problem should say ∣xi∣<1\lvert x_i\rvert<1.”
1911/16Silently assumes the negative part b<1b<1, but derives ∑∣xi∣−∣S∣=2min⁡(P,N)\sum\lvert x_i\rvert-\lvert S\rvert=2\min(P,N).
015/16Keeps the stated constraint: “−b<1-b<1 is always true because bb is positive.”
916/16Derives T−∣S∣=2min⁡(P,N)T-\lvert S\rvert=2\min(P,N) without changing the constraint.
AuthorsOpenReview ↗

Response to Reviewer dn8C (Part 2)

Response to Q1

See Response to W1.

Response to Q2

Yes, we provide the Table 3,4,5 results for the other two models below, all averaged over three evaluation runs.

For ablation study, we mainly performed no policy refinement for the other two models.

MethodA25Olym-EOlym-HHMMT25HMMT26Avg.Tok.×
R1-Distill-Llama-8B: Single33.336.05.030.021.225.11.00
R1-Distill-Llama-8B: No policy refinement35.639.78.031.125.327.91.21
R1-Distill-Llama-8B: Full SSR37.842.710.332.229.330.51.40
MethodA25Olym-EOlym-HHMMT25HMMT26Avg.Tok.×
gpt-oss-20b: Single70.079.027.050.057.656.71.00
gpt-oss-20b: No policy refinement75.680.337.354.463.662.31.18
gpt-oss-20b: Full SSR78.981.745.758.968.766.81.28

SSR deployment behavior.

MetricR1-Distill-Llama-8Bgpt-oss-20b
Candidate rejection rate27.0%23.7%
Candidate attempts per committed step1.231.31
Committed steps per problem7.815.98
Gate evaluations per problem9.67.8
Steps reaching the MM-attempt cap0.7%0.9%
Problems exhausting the BB-rejection budget3.9%3.3%

Wall-clock time per evaluation run, with generation on six H200 GPUs.

MethodR1-Distill-Llama-8B wall-clock time (h)gpt-oss-20b wall-clock time (h)
Single pass0.270.13
SSR0.390.16
ESC (w=2w=2)1.590.56
Majority@81.860.79
USC1.910.80

We will add the results to the paper.

Reviewer_dn8COpenReview ↗

Thank You

Thank you for your detailed response. I have updated my score to reflect this.

Reviewer_hbSwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SSR that lets an LLM check its own reasoning using a lightweight LoRA adapter on its own backbone. When a step scores below a calibrated threshold, it’s thrown out and resampled from the same prefix. SSR improves accuracy on math benchmarks while being token efficient.

Critical strengths
  • SSR has high token efficiency at inference time where accuracy is comparable to maj@8 and PRM reranking at a fraction of their token cost.
  • Motivation is strong where the authors show 1600 rollouts per step are needed to detect a typical change, which justifies looking at the cumulative drop instead.
  • The generator stays frozen and the gate is a lightweight add-on, which is a practical design that does not change the base model’s behavior when the gate is off.
Critical weaknesses
  • The decoding procedure and the use of a same-backbone verifier closely follow prior step-level verification work (LATTS). The main novel contribution is confidence-qualified first-passage labeling relative to the root value, but the paper does not compare it against existing rollout-based labeling schemes (e.g., Math-Shepherd or OmegaPRM) under the same gate architecture, so its specific benefit is not isolated.
  • “Steps” are fixed-length chunks, not reasoning steps. A step is simply up to 2,048 tokens, which can cut mid-sentence. Performance collapses with short steps: 54.7% at 256 tokens versus 64.6% at 1,024. That’s a big unexplained cliff, and it suggests the method is sensitive to arbitrary chunking.
  • The scope is narrow. Everything is math with a checkable final answer, and the training pipeline depends on such a checker. It’s unclear how this transfers to code, proofs, or open-ended tasks.
  • The motivating evidence is thin in places. Figure 2(a) shows a single trajectory, and Figure 2(b) a single prefix with 27 candidates. Table 2 broadens this, but its confidence intervals are wide.
Questions for authors
  • Why does USC on R1-Distill score far below single pass? Choosing among 8 solutions should not be much worse than taking one.
  • What verifier did LATTS use? Is it a matched strong verifier that makes it a fair comparison to use?
  • Could you compare your confidence-qualified first-passage labeling method to existing rollout-based labeling schemes under the same gate architecture and matched rollout compute?
  • Could you show evidence that the labels are actually correct? And also report how much data abstention throws away.
5 comments
AuthorsOpenReview ↗

Response to Reviewer hbSw (Part 1)

We sincerely thank the reviewer for their careful review and valuable feedback.

Response to W1

Relation to LATTS. We agree that LATTS and SSR share a high-level accept-or-resample decoding skeleton. However, our setting is substantially different from LATTS. LATTS evaluates only Llama-3.2-Instruct-1B/3B generators paired with larger external verifiers: primarily a Qwen2.5-7B PRM and additionally a Llama-3.1-Instruct-70B critic. It therefore studies a strong external verifier guiding a weaker generator setting. LATTS does not train a verifier or evaluate whether its approach works for thinking models. In addition, no official implementation is released.

Why accept-or-resample is different for long CoTs. LATTS divides an instruct model’s solution into semantically delimited steps separated by double newlines. Its external PRM or critic scores whether each coherent step is correct or helpful. A rejected candidate is therefore treated primarily as a locally defective solution step that should be replaced. SSR addresses a different decision problem in long continous thinking. A locally incorrect-looking exploration may still be useful or may be corrected later by the generator. Thus, our gate does not predict whether a chunk is locally correct; it predicts whether committing that chunk causes a statistically resolved reduction in the generator’s probability of eventually reaching the correct answer. This policy-conditioned recoverability objective is absent from LATTS.

Relation to existing labelling methods. Math-Shepherd and OmegaPRM were developed for relatively short, instruction-style solutions, where a step is typically a semantically complete derivation, local correctness is assessable, and an introduced error tends to persist; correspondingly, OmegaPRM assumes that the continuation value is approximately monotonic and searches for the first erroneous step.

These assumptions do not hold for long CoTs, which average approximately 15K tokens and contain hypotheses, tentative calculations, self-corrections, and abandoned explorations. A locally “bad” segment may subsequently be repaired, and recoverability can decrease and later increase, so step correctness is not equivalent to whether the trajectory remains recoverable. Our measurements directly demonstrate this mismatch: On the training pool, 68% of OmegaPRM's trajectories end correct and receive no labels, and 75% of Math-Shepherd's hard labels are positive. Moreover, only 4–13% of adjacent value changes are statistically resolvable, with mean ∣ΔQ∣|\Delta Q| of merely 0.05–0.10. Thus, conventional stepwise labeling and monotonic first-error search provide sparse or unreliable supervision in the thinking setting, motivating our confidence-qualified, root-relative first-passage labels.

Response to W2

As discussed in our response to W1, semantic step boundaries are considerably less well-defined in long CoTs than in relatively short, instruction-style solutions. Existing methods such as LATTS split reasoning steps using double newlines (\n\n). However, this segmentation rule has several limitations for long-form thinking. A separate paragraph may contain only a brief sentence such as “Wait, ...” or a single math equation such as 5x+2y=105x+2y=10. The rule would treat each of these fragments as an independent reasoning step. In addition, it also produces an excessively large number of steps in long CoTs. This is also mentioned in literature: OmegaPRM explicitly treats “any sequence of consecutive tokens” as a valid step and observes that Math-Shepherd’s rule-based divisions often “lack semantic coherence,” concluding that semantically explicit cutting is unnecessary for PRM training.

The result at 256 tokens should not be interpreted as a performance collapse: its accuracy is essentially close to the single-pass baseline. Rather, short-step rejection provides no improvement because candidates sampled from the same prefix have no statistically identifiable difference in their downstream success probabilities.

We verify this through the candidate audit. For each fixed prefix, we sample C=8C=8 candidate steps of length ss, then estimate each candidate’s recoverability Q^c\hat Q_c using K=12K=12 independent continuations. The observed variance among the eight estimates contains both genuine between-candidate variation and finite-rollout Bernoulli noise. We estimate the noise-corrected candidate spread as SDtrue=max⁡(0, Var⁡c(Q^c)−1C∑cQ^c(1−Q^c)K).\mathrm{SD}_{\mathrm{true}}=\sqrt{\max\left(0,\,\operatorname{Var}_c(\hat Q_c)- \frac{1}{C}\sum_c\frac{\hat Q_c(1-\hat Q_c)}{K}\right)}. The results are:

Step lengthSDtrue\mathrm{SD}_{\mathrm{true}}
2560.000
5120.000
10240.126
20480.136
40960.187
AuthorsOpenReview ↗

Response to Reviewer hbSw (Part 2)

At 256–512 tokens, the true candidate spread is indistinguishable from zero at the resolution of our audit: the observed differences in Q^\hat Q can be entirely explained by rollout noise. Consequently, there is no reliable candidate-level signal for the gate to learn. From 1024 tokens onward, candidates begin to make sufficiently consequential reasoning commitments that their recoverabilities measurably diverge, enabling effective gate learning and selection.

We will add the above explanation to the paper.

Response to W3

We agree the evaluation is limited to math. The method requires a binary outcome signal only at training time, to label recoverability; the deployed gate needs no checker. It therefore applies directly wherever outcomes can be verified automatically: code with unit tests, formal proofs with a proof checker, and science QA with reference answers. Informal proofs and open-ended generation lack a reliable checker; using a learned judge as the outcome signal would introduce label noise. We leave this as future research direction.

Response to W4

Figures 2(a) and 2(b) are intended only as illustrative examples; the corresponding aggregate evidence is reported in Table 2.

To reduce confidence intervals, we re-ran the characterization at a larger scale and under one protocol for all three models: 100 test problems per model (20 per benchmark), unselected trajectories, 5-endpoint chains with 16 rollouts per endpoint for adjacent changes, and 8 candidates per prefix with 32 rollouts each. Intervals are 95% problem-level bootstrap CIs.

Model∣ΔQ∣\lvert \Delta Q \rvertresolved@16Cand. SDTail
Qwen3-14B0.049 [0.036, 0.064]4.0% [1.8, 6.7]0.108 [0.075, 0.136]13.0% [8.5, 18.0]
R1-Distill-Llama-8B0.060 [0.037, 0.087]7.7% [4.4, 11.6]0.099 [0.071, 0.126]13.4% [8.3, 19.3]
gpt-oss-20b0.104 [0.083, 0.127]12.6% [8.7, 16.7]0.150 [0.107, 0.185]22.1% [16.3, 28.2]

The intervals are substantially narrower, and both conclusions of §2 hold with these bounds:

  1. Adjacent changes are rarely resolvable. At most 16.7% of adjacent differences are resolved at the upper CI bound, so per-step value targets are dominated by noise.
  2. Candidates sampled from the same prefix differ beyond Monte Carlo noise. Noise-corrected SD is 0.10–0.15, and a minority (13–22%) of candidates falls in the low-value tail, so selective rejection has something to act on.

Response to Q1

In USC, the model needs to read summaries of eight candidate solutions and output the index of the selected one in the format k\boxed{k}. R1-Distill-8B failed to produce a parseable index on 46% of the problems. We therefore attribute this result to an instruction-following failure and score those cases as incorrect.

Response to Q2

  1. In the original LATTS paper, the primary verifier is Qwen2.5-Math-PRM-7B; the authors also evaluate Llama-3.1-70B-Instruct as a step-level critic. The generators are Llama-3.2-Instruct-1B and 3B. Thus, LATTS operates in a strong verifier with a weak generator setting, where the verifier is substantially larger than the generator. Its implementation has not been publicly released.

  2. The PRM used in the original paper is inapplicable to our setting. Qwen2.5-Math-PRM-7B has a context window of only 4K tokens, whereas the median length of our reasoning trajectories ranges from 7K to 17K tokens.

  3. We use Skywork-Reward-V2-Qwen3-8B, which has a context window of 40960 tokens. This is a strong verifier: when used for ORM reranking, it improves over single-pass decoding by 6.8, 7.4, and 11.0 percentage points for Qwen3-14B, R1-Distill-Llama-8B, and gpt-oss-20b, respectively.

  4. We additionally evaluate LATTS with Qwen3-32B as the critic on Qwen3-14B, averaged over three runs:

MethodA25Olym-EOlym-HHMMT25HMMT26Macro Avg.Tok×
Single pass70.083.021.050.048.554.51.00
LATTS (Skywork-Reward-V2-Qwen3-8B)70.080.033.343.354.556.21.14
LATTS (Qwen3-32B critic)70.082.025.046.751.555.01.35
SSR77.895.735.062.252.564.61.21

we will add the results to the paper.

AuthorsOpenReview ↗

Response to Reviewer hbSw (Part 3)

Response to Q3

Due to time constraints, we primarily compared our labeling method with Math-Shepherd (soft labeling) and OmegaPRM on Qwen3-14B using SFT only.

The three labeling methods, SSR, Math-Shepherd, and OmegaPRM, use the same gate architecture and training objective, with matched labelling-rollout-generation token budgets. Only the labels differ. All methods are trained on the same set of Qwen3-14B training problems.

At inference time, the three methods use exactly the same accept-or-resample decoder and decoding hyperparameters; only the LoRA adapter is changed. The threshold η\eta is calibrated separately for each adapter so that all methods achieve the same target rejection rate of 17.8%.

Math-Shepherd

  1. Trajectories: Up to four trajectories per problem (j=0,…,3j=0,\ldots,3).

  2. Rollouts: At every position in each trajectory, we sample N=8N=8 rollouts.

  3. Labels: Every position is labeled.

    • The soft label is the fraction of the eight rollouts that produce a correct answer, taking values in {0,1/8,…,10,1/8,\ldots,1}.
    • The hard label is 1 if at least one of the eight rollouts is correct and 0 otherwise.
  4. Training objective: The soft label is used directly as the BCE target yy:

−[ylog⁡σ(z)+w(1−y)log⁡(1−σ(z))],w=∑y∑(1−y).-\left[y\log\sigma(z)+w(1-y)\log(1-\sigma(z))\right],\qquad w=\frac{\sum y}{\sum(1-y)}.

  1. Cost: The cost of a trajectory includes the tokens used to generate its steps and the tokens generated by the eight rollouts at every position.

OmegaPRM

  1. Trajectories: Up to eight trajectories per problem (j=0,…,7j=0,\ldots,7). We first evaluate the final answer of each complete trajectory.

    • Correct trajectories are not searched and produce no labels, although the tokens used to generate them are still included in the cost. The original method likewise searches only incorrect rollouts.
  2. Binary search: Each probe uses k=8k=8 rollouts, with c>0c>0 as the decision criterion.

    • Let last denote the final step. Because the complete solution is already known to be incorrect, we set V^(last)=0\hat V(\text{last})=0 without performing additional rollouts.
    • Starting with lo = 0 and hi = last, each iteration sets mid=⌊lo+hi2⌋\text{mid}=\left\lfloor\frac{\text{lo}+\text{hi}}{2}\right\rfloor and performs eight rollouts from mid.
    • If at least one rollout is correct (V^>0\hat V>0), the error is assumed to occur later, and we set lo = mid + 1.
    • If all eight rollouts are incorrect, the error is assumed to occur at or before mid, and we set hi = mid.
    • The search terminates when lo = hi, with lo identified as the first erroneous step.
    • This requires ⌈log⁡2T⌉\lceil\log_2 T\rceil probes for a TT-step trajectory, approximately three to four probes in our setting.
  3. Labels: Only probed positions are labeled.

    • For probed positions before the first erroneous step with V^>0\hat V>0, the label is V^\hat V, corresponding to the pointwise soft Monte Carlo target that performed best in the original paper.
    • The first erroneous step receives label 0.
    • Positions after the first erroneous step are left unlabeled, consistent with the original method.
    • If every probe has V^>0\hat V>0, the first error is assigned to the final step, which receives label 0.
  4. Training objective: We use the same weighted BCE objective as for Math-Shepherd.

  5. Cost: The cost of a trajectory includes the tokens used to generate its steps and the rollout tokens generated at the probed positions.

Observations: On the training pool, 68% of OmegaPRM trajectories end with a correct answer and therefore receive no labels. Meanwhile, 75% of Math-Shepherd’s hard labels are positive.

Gate architecture: All three gates use exactly the same configuration:

  • The same rank-16 LoRA adapter with the LM head frozen and the same +/−+/- token readout;
  • The same pointwise class-weighted BCE objective;
  • The same hyperparameters and training epochs.

The results across three runs:

MethodA25Olym-EOlym-HHMMT25HMMT26Avg.Tok.×
Single pass70.083.021.050.048.554.51.00
Math-Shepherd soft (BCE)70.083.721.751.148.555.01.45
OmegaPRM (BCE)68.985.022.352.245.554.81.35
SSR (BCE)72.286.024.353.349.557.11.13

It can be seen that SSR outperforms the baselines which are close to single pass.

We will add the results to the paper.

Reviewer_tz83OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Self-Step Rejection (SSR), a training and inference framework that enables a language model to selectively reject low-quality reasoning steps and resample alternatives from the same reasoning prefix. Unlike conventional approaches that rely on external process reward models or generate multiple complete solutions, SSR trains a lightweight LoRA-based acceptance gate on the model's own backbone. The gate is supervised using a first-passage labeling strategy based on estimated continuation success probabilities and is further optimized through group-relative policy refinement. During inference, the model evaluates each newly generated reasoning step and either accepts it or resamples a replacement, subject to predefined computational budgets. Experiments across three reasoning models and five mathematical reasoning benchmarks show consistent accuracy improvements over single-pass generation, with reported gains of 5.4–10.1 percentage points at approximately 1.21–1.40× the generation-token cost. The results suggest that selective step-level rejection may offer an effective trade-off between reasoning accuracy and inference-time computation.

Critical strengths
  • The paper addresses an important limitation of existing inference-time scaling methods, which is the inability to efficiently identify and replace locally flawed reasoning steps without regenerating entire solutions or relying on external verifiers
  • The paper provides an interesting methodological design. The first-passage labeling strategy is a principled way to construct step-level acceptance and rejection labels from trajectory-level correctness signals.
  • The method improves accuracy across three different reasoning models and five mathematical benchmarks. The results seem solid.
Critical weaknesses
  • Constructing the acceptance gate requires numerous continuation rollouts to estimate recoverability. I suppose the training of the acceptance gate is very expensive when extended to larger models.
Questions for authors
  • How well does the proposed method scale to larger language models?
  • Does SSR retain its effectiveness when applied to non-mathematical reasoning tasks or to models with different architectures and training procedures?
1 comment
AuthorsOpenReview ↗

Response to Reviewer tz83

We sincerely thank the reviewer for their careful review and valuable feedback.

Response to W1

We agree that continuation generation is the main training-time cost when scaling SSR to larger models. The gate itself remains a LoRA adapter, and no rollout or checker is required after deployment; nevertheless, generating policy-matched continuations becomes more expensive as the frozen generator grows.

Our current data-collection procedure already avoids dense, fixed-budget evaluation: (i) rollout counts are increased adaptively only when the confidence interval remains unresolved; (ii) the root estimate is reused from prescreening; (iii) labeling terminates at the first confidence-resolved barrier crossing; and (iv) same-prefix candidate enrichment is performed only at the crossing position. Thus, we concentrate rollout computation near uncertain decisions rather than applying the maximum budget to every step.

We have evaluated models up to 20B parameters. For larger generators, several techniques may help further reduce the cost. First, a smaller rollout policy may replace the generator when its estimates are sufficiently aligned: MLR [1] shows that a small rollout model with TSMC importance weighting can provide future-solvability estimates on some tasks. Second, tree-structured sampling can reuse shared prefixes and rollout outcomes instead of independently restarting Monte Carlo estimation at every step. SPO-tree [2] explicitly reports that node sharing and sample reuse substantially reduce sampling overhead, while TreeRL [3] branches selectively from high-uncertainty intermediate states under a fixed generation budget.

We view adapting these techniques to SSR labeling as promising future work.

Response to Q1

Due to the time constraint, we will investigate scalability to larger models in future work.

Response to Q2

Our method requires a binary outcome signal only at training time, to label recoverability; the deployed gate needs no checker. It therefore applies directly wherever outcomes can be verified automatically: code with unit tests, formal proofs with a proof checker, and science QA with reference answers. Due to the time constraint, we will investigate these tasks in future work.

Different architectures and training procedures. Our three generators already differ along both axes:

  • Qwen3-14B is a dense model with a native thinking mode, distilled from larger Qwen3 models;
  • R1-Distill-Llama-8B is a dense Llama-3.1 model trained only by supervised distillation from DeepSeek-R1 (no RL);
  • gpt-oss-20b is a mixture-of-experts model trained with reinforcement learning, using the Harmony format with a separate reasoning channel.

SSR improves macro-average accuracy on all three at 1.21–1.40× single-pass tokens, and it is the strongest step-level method for each (Table 1).

Reference

[1] Xiong, Siheng, Ali Payani, and Faramarz Fekri. "Enhancing language model reasoning with structured multi-level modeling." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Guo, Yiran, et al. "Segment policy optimization: Effective segment-level credit assignment in rl for large language models." Advances in Neural Information Processing Systems 38 (2026): 114399-114431.

[3] Hou, Zhenyu, et al. "Treerl: Llm reinforcement learning with on-policy tree search." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.