acceptodds
Under review as a conference paper at ICLR 2027

Learning Process Rewards via Reasoning State Propagation

Abstract

Process reward models (PRMs) have demonstrated notable effectiveness in test-time scaling and reinforcement learning by providing fine-grained signals for evaluating intermediate reasoning states, but their training relies heavily on costly process annotations. A natural way to alleviate this dependence is to complement limited process supervision with scalable outcome supervision. However, existing PRMs often model reasoning prefixes independently, providing no explicit mechanism for effectively using final outcome to guide the learning of intermediate reasoning states. We introduce Reasoning State Propagation (RSP), which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory. Specifically, RSP predicts a break probability that a valid state becomes invalid and a repair probability that an invalid state returns to valid. By propagating these transitions, RSP connects intermediate states to the final state, allowing process annotations to supervise intermediate states while outcome labels supervise the final state and provide learning signals to preceding steps. Across reasoning search, response selection, and reinforcement learning, RSP consistently outperforms representative PRM baselines, with average improvements over Qwen2.5-Math-PRM of 5.6% in beam search and 2.1% in reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Process reward model, Semi-supervised learning, Credit assignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_R9a6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reasoning State Propagation (RSP) which uses outcome labels to guide the training of PRMs. The authors first introduce the repair probability to capture the state when the LLMs resolve earlier errors. And they define a state propagation matrix which allows the outcome supervision to guide the process states. Then, PRM can be trained with less reliance on costly process annotations. Experiment results shown that the proposed RSP outperforms other compared methods on different datasets and LLMs.

Critical strengths

1.The motivation is clear and valuable.RSP explores how to leverage low-cost scalable outcome supervision during PRM training.

2.This paper introduce a new repair probability to capture the self-correction step which is reasonable for current reasoning LLMs.

3.The experiments cover multiple test-time scaling methods and reinforcement learning to show the effectiveness of the proposed RSP.

4.RSP outperforms most PRMs on different datasets and LLMs.

Critical weaknesses

1.Table.1 lacks the results compared with Qwen2.5-Math-PRM. Authors should add the experiments such as Qwen2.5-Math-7B with RSP or Qwen3 with Qwen2.5-Math-PRM.

2.Although the state propagation matrix allows the final state to be computed from previous state transitions, Eq.5 can not identifies which steps introduce or correct errors. A correct final answer can cause the model to decrease all break probabilities and increase all repair probabilities.

3.RSP uses PRM800K as the process-annotated training set but it contains no B->G labels. How does RSP learns to identify repair steps?

Questions for authors

See Weaknesses

3 comments

Dear Reviewer R9a6,

Thank you for recognizing the motivation, repair formulation, and downstream evaluation. We clarify the comparison in Table 1 and explain how repair is trained and evaluated.

Weakness #1: Comparison with Qwen2.5-Math-PRM in Table 1

Following the reviewer’s suggestion, we additionally train RSP using Qwen2.5-Math-7B as the backbone and compare it with Qwen2.5-Math-PRM-7B. The results are shown below.

ModelBeam SearchBoN
Qwen2.5-Math-PRM-7B64.874.1
RSP (Qwen2.5-Math-7B backbone)69.474.5

RSP achieves higher accuracy in both settings, improving beam search by 4.6 percentage points and BoN by 0.4 percentage points over Qwen2.5-Math-PRM-7B. The alternative comparison using Qwen3 with Qwen2.5-Math-PRM cannot be conducted under matched training conditions, as the training data used for Qwen2.5-Math-PRM have not been publicly released. We will include the above results in the revised version.

Weakness #2: Whether a correct outcome simply lowers all break scores and raises all repair scores

Eq. (5) defines state propagation, while the learning objectives in Eqs. (8) and (9) train the transitions using process and outcome labels. Process labels constrain intermediate validity, and outcome labels train preceding transitions through the final state. This joint training is essential to interpreting the learned break and repair scores.

Appendix C.3 gives the outcome gradients with respect to the break and repair logits:

=\frac{p_{t-1}^G\alpha_t(1-\alpha_t)}{Z_{\mathrm{out}}} \bigl(h_t(G)-h_t(B)\bigr),$$ $$\frac{\partial\ell_{\mathrm{out}}}{\partial\eta_t^{\mathrm{rp}}} =-\frac{p_{t-1}^B\beta_t(1-\beta_t)}{Z_{\mathrm{out}}} \bigl(h_t(G)-h_t(B)\bigr).$$ Here, $h_t(a)$ is the probability of reaching the observed terminal target from state $a$ through the remaining steps, and $Z_{\mathrm{out}}$ is the probability of that target. The gradients depend on the preceding state and the subsequent transitions. In particular, the repair gradient is weighted by $p_{t-1}^B$, and both gradients vanish when the remaining transitions make the terminal target equally likely from G and B. When $h_t(G)>h_t(B)$, the outcome loss favors lower break and higher repair at that position. These directions can agree across steps, but the update magnitudes depend on the preceding states and remaining transitions. Process labels also constrain intermediate validity, including invalid states, so predicting every prefix as valid would conflict with those annotations. We further evaluate whether RSP can distinguish error recovery from error persistence in our response to Weakness #3 you proposed, so you can find more details below.

Weakness #3: Learning repair without B→G labels in PRM800K

PRM800K primarily supervises states before and at the first error. Repair is therefore learned indirectly from complete outcome-annotated trajectories, using the same model trained on process labels. If an earlier prefix is predicted invalid but the final outcome is correct, fitting both predictions requires the remaining transitions to allow recovery. Appendix C.3 formalizes this: when pkB≥1−ϵp_k^B\geq1-\epsilon and pTG≥1−δp_T^G\geq1-\delta, hk(B)≥1−δ1−ϵ.h_k(B)\geq1-\frac{\delta}{1-\epsilon}. Here, hk(B)h_k(B) is the probability of reaching a valid terminal state from B through the remaining steps. Conditional on preserving the invalid prefix prediction, the positive outcome requires sufficient recovery in those steps. This explains how repair receives supervision without explicit B→G labels. The terminal supervision constrains the cumulative probability of recovery over the remaining trajectory, while the transition predictors parameterize how this constraint is distributed across subsequent reasoning steps. Whether these transitions align with actual recovery events is further examined in our recovery evaluation below.

To evaluate that behavior, we use GPT-6.1 Sol to annotate Qwen3-1.7B solutions to DAPO-Math-17K questions. Labels indicate whether the current prefix contains an unresolved error, including after the first error. A locally correct new step is not a repair if the earlier error still affects the current derivation. We evaluate 19,111 adjacent pairs with a preceding B state, including 3,376 B→G and 15,735 B→B pairs. Both models score the same pairs without retraining. F1 uses a threshold of 0.5.

MethodAUROCAUPRCF1
Qwen2.5-Math-PRM-7B0.70530.26590.4086
RSP0.77880.37940.4616

RSP performs better on all three metrics, supporting its ability to distinguish recovery from error persistence under these annotations.

We thank the reviewer for this suggestion. It also inspires us to further improve RSP by incorporating directly annotated B→G examples into training. In fact, collecting such data is feasible with strong models such as GPT-6. We believe that such data could further improve RSP by providing direct supervision for repair transitions. We plan to explore this in future work.

We have also uploaded a revised version of the manuscript, in which all corresponding revisions are clearly marked at their respective locations for ease of reference.

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Reviewer_8JY7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Reasoning State Propagation (RSP), which represents each reasoning prefix as a distribution over valid and invalid states. Two transition predictors estimate the probability of introducing an error and the probability of repairing an unresolved error. These predictions form transition matrices that recursively update the state distribution. Process labels supervise intermediate states, while outcome labels supervise the final propagated state, providing gradients to preceding transitions.

Critical strengths

RSP provides an explicit, differentiable relationship between intermediate validity states and the terminal state. Its repair transition can represent reasoning that recovers after an earlier mistake, and the accompanying analysis explains how both supervision sources train the transition predictors. This is consequential because it offers a concrete mechanism for using abundant outcome labels alongside limited process annotations, addressing a practical bottleneck in PRM training.

Critical weaknesses
  1. The modeling approach offers limited novelty and appears incremental.

  2. The consistency between outcome labels and reasoning validity requires further validation. Although the training data have been filtered by a judge, a correct answer does not necessarily imply that reasoning errors have been resolved. The paper does not quantify the agreement between answer correctness and terminal reasoning validity after filtering, leaving unclear whether potential label mismatches affect state transition learning.

  3. The impact of outcome supervision on earlier steps remains unclear. Gradients may decay rapidly along the explicit state propagation chain, so the gains may primarily arise from local effects near the end of the trajectory. Ablations with different backpropagation horizons are needed to assess whether earlier steps receive effective supervision.

Questions for authors

PRM800K’s annotation protocol primarily covers valid states before the first error and the first error itself, providing limited direct supervision for repair. Although outcome supervision can train repair indirectly, there is insufficient evidence that the model learns actual error recovery rather than merely adjusting state scores to fit the final labels.

6 comments

Dear Reviewer 8JY7,

Thank you for recognizing the value of connecting intermediate reasoning states to outcome supervision. We address the novelty, validity, and gradient concerns below.

Weakness #1: Novelty of the modeling approach

TWeakness #1: Novelty of the modeling approach

The main novelty of RSP lies in its reasoning state propagation formulation, which provides an explicit and recoverable representation of reasoning validity and establishes a differentiable connection between intermediate reasoning states and final outcomes. Rather than simply adding outcome supervision to an existing PRM, RSP changes how reasoning validity is modeled and how process and outcome supervision are connected. Specifically, this contribution and novelty are reflected in the following four aspects:

  1. Explicit state transitions rather than independent prefix predictions. Conventional discriminative PRMs predict the validity of individual reasoning prefixes without explicitly modeling how reasoning validity evolves across successive steps. Although some existing methods introduce sequential dependencies, they do not fully capture flexible transitions between valid and invalid states. RSP represents each prefix using a probability distribution over two reasoning states and models the transition between adjacent states through break and repair probabilities. Consequently, the current state depends explicitly on both the preceding state and the newly incorporated reasoning step, providing a structured representation of validity evolution throughout the trajectory.
  2. Recoverable reasoning states beyond first-error modeling. Existing approaches such as CRM primarily model the first transition into an invalid state and effectively treat subsequent reasoning as remaining invalid. RSP additionally models the transition from an invalid state back to a valid state through the repair probability. This allows the framework to represent both error introduction and subsequent recovery, including trajectories in which earlier errors are corrected or superseded by later reasoning. Thus, RSP does not impose a monotonic relationship between intermediate validity and final correctness.
  3. A unified mechanism for process and outcome supervision. The key distinction from simply adding an outcome loss to an existing PRM is how the two supervision sources are connected. In Joint-Supervised PRM, outcome supervision is applied only to the independently predicted final prefix score. In contrast, RSP computes the final state through the sequential propagation of intermediate state transitions. Process annotations directly supervise intermediate propagated states, while outcome labels supervise the terminal state and backpropagate through the propagation chain to preceding break and repair predictions. This provides an explicit mechanism through which scalable outcome supervision contributes to intermediate state learning.
  4. Experimental evidence for the contribution of state propagation. We conduct a matched comparison with a Direct-State Classifier that uses the same backbone, special tokens, adjacent boundary representations, MLP heads, training data, and optimization settings as RSP, but removes state dependence by directly predicting prefix validity. RSP improves over this matched baseline on beam search (72.2 vs. 68.7), Best-of-N selection (74.4 vs. 73.6), and ProcessBench (65.8 vs. 63.9). This comparison provides evidence that the gains are not merely attributable to the boundary representations or prediction heads, but also to explicit state propagation. Furthermore, removing repair or blocking outcome gradients to preceding transitions consistently degrades performance, supporting the respective contributions of recoverable state transitions and outcome supervision through propagation.

Overall, RSP's contribution is not any individual transition probability or supervision loss in isolation, but their integration into a unified state propagation formulation that enables recoverable reasoning-state modeling and joint learning from intermediate process annotations and terminal outcome labels. We will include these analyses in the revised manuscript.

Weakness #2: Agreement between outcome labels and terminal reasoning validity

A correct answer can coexist with unresolved reasoning errors, which is why we use filtered outcome data and audit its agreement with terminal validity. In RSP, G means that no unresolved error remains in the current derivation. Earlier errors may have been corrected or superseded. Outcome labels supervise only the final propagated state, rather than assigning the final label to every intermediate step.

As described in Appendix A.4, AceMath-RM verifies answers with the Qwen-Math evaluation toolkit, then uses Qwen2.5-Math-RM-72B ranking and score-sorted sampling to retain high-confidence responses. To quantify agreement after filtering, we audit 5,000 outcome-positive AceMath-RM trajectories using GPT-6.1 Sol. The judge checks whether the derivation retained at the end supports the final answer without an unresolved error. A correct answer does not suffice if its supporting derivation remains faulty. About 93.4% of the sampled trajectories were judged to have valid terminal states.

This provides quantitative support for using filtered positive outcome labels as terminal supervision. We will further identify and remove noisy samples and examine how improved filtering affects the learned rewards.

Weakness #3: Outcome supervision of earlier transitions

Outcome supervision trains earlier transitions through the state propagation chain. As shown in Appendix C.4,

=\prod_{j=t+1}^{T}(1-\alpha_j-\beta_j),$$ so the learned transitions determine how strongly the final state depends on earlier states. We examine this dependence through both an ablation and direct gradient measurements. Our No Outcome Propagation ablation computes the outcome loss with $p_T=\operatorname{sg}(p_{T-1})M_T$. It preserves forward propagation and outcome gradients to the final transition while blocking the path through the recurrence to earlier transition outputs. Process supervision is unchanged. This reduces beam search accuracy from 72.2 to 65.5 and ProcessBench F1 from 65.8 to 61.8. Thus, allowing outcome gradients through earlier transitions improves the evaluated results. This comparison covers full propagation and retaining outcome gradients only at the final transition. Experiments with additional backpropagation horizons are not yet available. We further measure the magnitude of the outcome-loss gradient with respect to the break and repair logits on **1,024 AceMath-RM trajectories** at the trained checkpoint. Parameters are fixed, and gradients are computed independently for each trajectory without process losses. For each trajectory, we find the earliest transition whose gradient magnitude is at least a given fraction of that trajectory's maximum. Its distance from the end, divided by $T-1$, is the normalized reach. | Relative gradient threshold | Median normalized reach | Reaching midpoint or earlier | Reaching first transition | | :-------------------------: | :---------------------: | :--------------------------: | :-----------------------: | | 10% | 16.7% | 6.2% | 0.4% | | 5% | 25.0% | 16.8% | 3.0% | | 1% | 33.3% | 31.0% | 7.3% | | 0.1% | 37.5% | 37.8% | 10.3% | These measurements show that outcome gradients reach beyond the final transition, with the distance varying across trajectories. Gradients are concentrated near the end in many cases and extend further in others. Equal supervision strength at every position is not required for outcome data to improve intermediate-state learning. The ablation and gradient measurements support this contribution. Long-range gradient decay remains a direction for improvement. We will explore directly supervising the probability of reaching the final state from each intermediate state using outcome labels. We will include these analyses in the revised manuscript.

Question: Learning actual recovery without direct B→G labels in PRM800K

PRM800K trains prefix validity before and at the first error. Outcome supervision then trains the same transition predictors on complete trajectories, including those whose errors are followed by recovery. This is the role of combining the two supervision sources when direct B→G annotations are scarce.

Appendix C.3 makes this connection precise. If an intermediate prefix is predicted invalid with pkB≥1−ϵp_k^B\geq1-\epsilon and a positive outcome is fitted with pTG≥1−δp_T^G\geq1-\delta, then hk(B)≥1−δ1−ϵ,h_k(B)\geq1-\frac{\delta}{1-\epsilon}, where hk(B)h_k(B) is the probability of reaching a valid terminal state from B through the remaining steps. Conditional on retaining the invalid-prefix prediction, fitting the positive outcome requires recovery in the remaining reasoning. The outcome constrains the combined effect of later transitions, while the location of recovery is learned from the reasoning context. We evaluate that location through adjacent-state annotations below.

We therefore test recovery directly using Qwen3-1.7B solutions to DAPO-Math-17K questions, annotated by GPT-6.1 Sol. The judge checks whether the current prefix contains an unresolved error and continues after the first error. A locally correct step is not labeled G if the earlier error still affects the current derivation. The shared evaluation contains 19,111 pairs with a preceding B state: 3,376 B→G and 15,735 B→B, from 7,371 trajectories and 2,311 questions. Both models are evaluated without retraining, with a threshold of 0.5 for F1.

MethodAUROCAUPRCF1
Qwen2.5-Math-PRM-7B0.70530.26590.4086
RSP0.77880.37940.4616

RSP outperforms Qwen PRM on all three metrics. This evaluation tests recovery against continued error using prefix annotations, providing evidence beyond fitting final labels.

We thank the reviewer for this suggestion. It also inspires us to further improve RSP by incorporating directly annotated B→G examples into training. As our evaluation suggests, collecting such data is feasible with strong models such as GPT-6. We plan to explore this in future work.

We have also uploaded a revised version of the manuscript, in which all corresponding revisions are clearly marked at their respective locations for ease of reference.

Reviewer_d3M4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Reasoning State Propagation (RSP), a process reward modeling framework that combines process annotations with outcome supervision. RSP represents each reasoning prefix using a binary validity state and predicts two transition probabilities: break, for moving from a valid to an invalid state, and repair, for returning from an invalid to a valid state. Process annotations supervise intermediate state probabilities, while outcome labels supervise the terminal state through the propagation chain. Experiments evaluate the resulting rewards in beam search, Best-of-N selection, reinforcement learning, and process error identification.

The main contribution is a recoverable-state formulation of process rewards, together with an empirical study under mixed supervision. The strongest results are in beam search. The mathematical formulation is coherent, although the interpretation of the learned transitions and the robustness of some downstream comparisons require further evidence.

Critical strengths
  1. The beam-search improvements are substantial and reasonably consistent. In Table 1, RSP with the Qwen3-4B backbone achieves the best performance in all six combinations of benchmark and expansion width, averaging approximately 5.6 percentage points above Qwen2.5-Math-PRM-7B. More importantly for attribution, it also improves over Joint-Supervised PRM, which uses the same backbone, training-data mixture, and optimization settings. This comparison provides evidence that the gains are not explained solely by access to additional outcome-labeled data.

  2. The evaluation and ablations cover several relevant aspects of the method. The paper studies search, response selection, policy optimization, and error identification. Table 4 examines supervision sources, repair, boundary representations, token parameterization, and outcome-gradient propagation. In particular, blocking outcome gradients to earlier transitions reduces beam-search accuracy by 6.7 percentage points and ProcessBench F1 by 4.0 points while retaining forward propagation. This provides meaningful evidence for the training contribution of the recursive structure. The RL comparisons also explicitly control policy initialization, training data, and optimization budget.

Critical weaknesses

1. The empirical validation of recovery detection and the attribution of gains to explicit state propagation remain incomplete.

Proposition 1 provides a conditional link between accurate adjacent-state predictions and the corresponding transition probabilities, and Appendix F presents qualitative recovery examples. However, these do not establish the accuracy or calibration of recovery predictions on independently annotated trajectories.

Section 4.5 evaluates transition scores using approximately 10K step pairs labeled with Qwen3-32B as the judge. The manuscript does not sufficiently document the source of these trajectories, their separation from training data, the number of pairs in each transition category, the judging instructions, or agreement with human annotations. The distinction between a locally correct new step and a step that resolves an earlier error is central here. Score-distribution plots provide limited evidence about this distinction, and ProcessBench evaluates the first error rather than recovery after an error.

The No Repair ablation supports allowing validity scores to recover, but disabling repair also imposes a monotonicity constraint on the propagated validity probability. Its degradation therefore does not, by itself, establish that RSP identifies genuine correction events. There is also a possible alternative explanation for the transition-score separation. From Equation (26), setting αt=1−qt\alpha_t=1-q_t and βt=qt\beta_t=q_t gives ptG=qtp_t^G=q_t, removing dependence on the preceding propagated probability. Such a predictor can still use the full textual prefix and could produce qualitatively similar separation between the transition categories if it predicts current-state validity accurately. This construction does not imply that the trained RSP model has collapsed to a reset solution. It motivates a matched direct-state classifier or forced-reset control to distinguish the contribution of explicit propagation from that of the boundary representations and prediction heads. Joint-Supervised PRM uses a different parameterization, while No Outcome Propagation retains forward propagation, so neither fully resolves this comparison.

Appendix C.4 raises a related question about propagation distance. The dependence through the state recurrence involves products of κt\kappa_t, while the reported median κt\kappa_t is 0.092 and 79.0% of transitions have ∣κt∣<0.2|\kappa_t|<0.2. The authors explicitly acknowledge that terminal supervision need not affect all preceding transitions equally. Nevertheless, the marginal statistics do not establish its effective propagation distance. Actual outcome-gradient magnitudes and propagation products, grouped by distance from the terminal step, would clarify this point.

Finally, using answer correctness to supervise terminal validity assumes sufficient agreement between two different notions of correctness. Appendix A.4 describes reasonable filtering but does not quantify this agreement. An audit of outcome-positive examples for unresolved reasoning errors would strengthen the interpretation of the repair signal.

2. The robustness of the downstream advantages remains insufficiently established.

The most concrete concern is the BoN scoring protocol. Section 4.3 uses the final step score for every PRM, whereas the original Qwen PRM evaluation uses the product of step scores (Zhang et al., 2025b, The Lessons of Developing Process Reward Models in Mathematical Reasoning, Section 2.2). Final-step scoring directly matches RSP’s terminal training objective, but its suitability for the other PRMs is not established. This matters because the average advantage over Qwen PRM in Table 2 is only 0.3 percentage points, 74.4 versus 74.1. Evaluating each baseline’s original scoring rule, together with alternatives selected on a held-out validation set, would establish whether the comparison is robust to aggregation choices. This concern applies directly to BoN; the available description does not establish that the same issue affects beam search.

The RL improvement also needs uncertainty estimates. Using the reported values in Table 3, the average gain over Qwen PRM is 2.1 percentage points, with the 7.8-point improvement on AIME contributing approximately 62% of the net gain across the six benchmarks. Relative to GRPO, the average change on the other five benchmarks is approximately −0.24 percentage points. The AIME result is valuable, but this concentration makes multiple training seeds and a precise description of the AIME/AMC evaluation particularly important.

The abstract would be more precise if it stated that the improvements concern average downstream performance, since RSP does not outperform every baseline for every generator in Table 2. Separately, its ProcessBench macro-average is below Qwen PRM, 65.8 versus 70.5, as the authors acknowledge in Appendix B. This limits a broader interpretation of the results as uniformly better process error identification, but does not directly contradict the abstract’s stated scope.

Questions for authors
  1. How does RSP compare with a direct-state classifier or forced-reset model using matched boundary representations and a comparable parameter budget? Please also specify the exact stop-gradient operation in No Outcome Propagation. Distance-resolved outcome-gradient statistics would help establish how far terminal supervision effectively propagates.

  2. How were the approximately 10K transition-analysis pairs sampled and labeled? What annotation criterion distinguishes a locally correct subsequent step from a genuine repair of an earlier error? Can you provide human-validated recovery-localization results and an estimate of agreement between answer correctness and terminal validity in the outcome data?

  3. Does the BoN advantage persist with each baseline’s original aggregation rule or aggregation selected on a held-out validation set? Please report results separately for each NN, clarify whether the reward models score identical candidate pools, and provide question-level paired confidence intervals.

  4. Which AIME and AMC versions and question sets are used, and how is Avg@16 computed? How stable are the AIME and aggregate RL improvements across independent training seeds?

6 comments

Dear Reviewer d3M4,

Thank you for the careful review and for recognizing the beam search improvements and the contribution of outcome propagation. We address your weaknesses and questions below.

Weakness #1 and Questions #1–2: Recovery detection, state propagation, and terminal validity

Recovery detection. To address whether RSP identifies recovery beyond the score distributions in Section 4.5 and the No Repair ablation, we add a quantitative evaluation of B→G versus B→B transitions. This directly tests the distinction between resolving an earlier error and continuing with an unresolved error.

We follow the trajectory construction procedure in the original analysis, replacing the Qwen3-32B judge with GPT-6.1 Sol (Medium request configuration). Qwen3-1.7B generates solutions to questions from DAPO-Math-17K. The judge labels whether the current reasoning prefix contains an unresolved error and provides a rationale for each label. A locally correct new step remains B if an earlier error still affects the current derivation. A B→G transition requires that the error be resolved or superseded by valid reasoning. The judge receives the question and the prefix through the target step, without the reference answer or existing labels, and continues annotating after the first error.

After filtering, the annotated data contain 9,593 trajectories from 2,488 questions and 56,709 steps. We retain adjacent pairs whose preceding state is labeled B. After excluding truncated trajectories and prefixes exceeding either model's context limit, the shared evaluation set contains 19,111 pairs from 7,371 trajectories and 2,311 questions, including 3,376 B→G pairs (17.67%) and 15,735 B→B pairs (82.33%). Both models are evaluated on the same pairs without retraining. F1 uses a threshold of 0.5.

MethodAUROCAUPRCF1
Qwen2.5-Math-PRM-7B0.70530.26590.4086
RSP0.77880.37940.4616

RSP outperforms Qwen PRM on all three metrics. These results provide quantitative evidence that RSP distinguishes error recovery from error persistence under the stated annotation criteria. Human validation at this scale could not be completed during the rebuttal period. We will further validate the transition labels and improve annotation quality.

Explicit propagation versus direct-state prediction. The No Repair ablation also imposes a monotonicity constraint on state validity, so we use the recovery evaluation above to assess actual B→G events. To separate the contribution of propagation from representations and prediction heads, we additionally compare RSP with a Matched Direct-State Classifier.

The original Joint-Supervised PRM comparison uses the same Qwen3-4B backbone, data mixture, and optimization settings. The parameter count increases only from 4.035B to 4.048B, approximately 0.3%, suggesting that the gains are unlikely to come simply from increased model capacity. The new direct-state comparison further matches the parameterization.

The Matched Direct-State Classifier uses the same backbone, BREAK and REPAIR tokens, adjacent boundary representations, and two MLP heads as RSP. It also uses the same training data, budget, and process and outcome supervision. We average the two head logits to predict current-state validity qtq_t and set αt=1−qt\alpha_t=1-q_t and βt=qt\beta_t=q_t during training and inference. This gives κt=1−αt−βt=0\kappa_t=1-\alpha_t-\beta_t=0 and ptG=qtp_t^G=q_t, removing dependence on the preceding propagated state while retaining access to the textual prefix.

MethodBeam SearchBoNProcessBench
Matched Direct-State Classifier68.773.663.9
RSP72.274.465.8

Beam Search averages MATH500 and Gaokao accuracy with b=4b=4 and e=8e=8. BoN follows Table 2, and ProcessBench reports macro-average F1. RSP improves all three results, supporting the benefit of explicit state propagation beyond the boundary representations and prediction heads.

Outcome-gradient propagation. In No Outcome Propagation, we compute the outcome loss using pT=sg⁡(pT−1)MT,p_T=\operatorname{sg}(p_{T-1})M_T, where sg⁡\operatorname{sg} stops gradients. This preserves forward propagation and outcome gradients to the final transition, while blocking the path through the state recurrence to earlier transition outputs. Process supervision is unchanged. Beam search accuracy drops from 72.2 to 65.5, and ProcessBench F1 drops from 65.8 to 61.8. These results show that training earlier transitions through outcome propagation improves process rewards.

To measure propagation distance, we analyze 1,024 sampled outcome-annotated AceMath-RM trajectories at the trained checkpoint. We keep model parameters fixed and compute gradients independently for each trajectory using automatic differentiation, excluding process losses. For trajectory ii, the gradient magnitude at transition tt is

\left(\frac{\partial\ell_{\mathrm{out}}}{\partial\eta_{i,t}^{\mathrm{br}}}\right)^2+ \left(\frac{\partial\ell_{\mathrm{out}}}{\partial\eta_{i,t}^{\mathrm{rp}}}\right)^2}.$$ Let $G_i^{\max}=\max_s G_{i,s}$ and define $$\mathcal S_i(\tau)=\{t:G_{i,t}\geq\tau G_i^{\max}\}.$$ The backward gradient reach and its normalized value are $$H_i(\tau)= \begin{cases} T_i-\min\mathcal S_i(\tau),&\mathcal S_i(\tau)\neq\varnothing,\\ -1,&\mathcal S_i(\tau)=\varnothing, \end{cases} \qquad \rho_i(\tau)=\frac{\max(H_i(\tau),0)}{T_i-1}.$$ Here, $H_i=0$ means that only the terminal transition qualifies, $H_i=1$ means the earliest qualifying transition is the penultimate one, and $H_i=T_i-1$ means the first transition qualifies. Intervening transitions need not all exceed the threshold. | Relative gradient threshold | Median normalized reach | Reaching midpoint or earlier | Reaching first transition | | :-------------------------: | :---------------------: | :--------------------------: | :-----------------------: | | 10% | 16.7% | 6.2% | 0.4% | | 5% | 25.0% | 16.8% | 3.0% | | 1% | 33.3% | 31.0% | 7.3% | | 0.1% | 37.5% | 37.8% | 10.3% | The results show that outcome supervision can reach earlier transitions, with its reach varying across trajectories. Gradients are concentrated near the end in many cases and extend further in others. The learned state transitions determine how far supervision propagates, rather than imposing a fixed distance. It is not necessary for the terminal loss to supervise every preceding transition equally strongly. Together with the No Outcome Propagation ablation, these measurements support the contribution of outcome supervision to intermediate-state learning. Our aim is to use scalable outcome data to complement limited process annotations, not to claim that long-range gradient decay is fully resolved. In future work, we will explore using outcome labels to directly supervise the probability of reaching the final state from each intermediate reasoning state. This may strengthen supervision of earlier transitions without relying only on gradients through subsequent transitions. **Outcome correctness and terminal validity.** In RSP, G means that no unresolved error remains in the current derivation. Earlier errors may have been corrected or superseded. Process labels supervise intermediate states where available, while outcome labels supervise only the final propagated state and provide gradients to preceding transitions. As described in Appendix A.4, AceMath-RM uses the **Qwen-Math evaluation toolkit** to verify answers, followed by ranking with **Qwen2.5-Math-RM-72B** and score-sorted sampling to retain high-confidence responses. To quantify agreement between the filtered positive outcome labels and terminal validity, we audit **5,000 outcome-positive trajectories using GPT-6.1 Sol**. The judge checks whether the derivation retained at the end supports the answer without an unresolved error. Earlier errors may be corrected, replaced, or abandoned, but a correct answer alone is insufficient if its supporting derivation remains faulty. **About 93.4% of the sampled trajectories were judged to have valid terminal states.** This supports using filtered positive outcome labels to supervise the final propagated state. We will further identify and remove noisy samples and examine how improved filtering affects RSP's performance. The full GPT-6.1 Sol evaluation prompt is provided in Appendix A.4 of our revised manuscript.

Weakness #2 and Questions #3–4: Downstream robustness and evaluation settings

BoN aggregation and candidate pools. We use final-step scores for all PRMs as a common protocol. For RSP, this also matches the training objective that supervises the final propagated state. To test whether the advantage depends on this choice, we evaluate Product, Min, Mean, and Final aggregation on the same candidate responses. Product includes the original Qwen PRM scoring rule identified in the review.

MethodProductMinMeanFinal
Qwen2.5-Math-PRM-7B73.874.173.974.1
RSP74.173.974.774.4

RSP performs better under Product, Mean, and Final, while Qwen PRM performs slightly better under Min. RSP achieves the best result in this comparison under Mean aggregation (74.7%). These results show that RSP's advantage extends beyond final-step scoring. We will include these comparisons in the revised manuscript.

All reward models score exactly the same candidates for each question and NN. With Final aggregation, the results averaged across generators are:

MethodN=8N=16N=32N=64N=128
Qwen2.5-Math-PRM-7B72.172.974.375.275.9
RSP72.373.374.375.676.5

RSP matches or exceeds Qwen PRM at each evaluated NN. Using 1,000 paired bootstrap resamples over questions, the 95% confidence interval for the overall accuracy difference, averaged across generators and the five NN values, is about [0.08, 0.52] percentage points. This supports the average BoN improvement under identical candidate pools. We will include these results in the revised manuscript.

RL settings and stability. All RL methods use the same policy initialization, training data, and optimization budget. Although AIME contributes substantially to the overall gain, the improvement over Qwen PRM also extends to AMC, MATH500, and Olympiad.

We use AIME 2024 (AIME-24) and AMC 2023 (AMC23), following the benchmark setting of VeriGate. Avg@16 is the average accuracy over 16 independently sampled responses per question. The overall Avg is the arithmetic mean across six benchmarks.

Due to the limited rebuttal time and computational budget, we have not completed additional independent training seeds. We will prioritize these experiments and include results from multiple training seeds and uncertainty estimates in the revised version before publication.

Scope of the claims and ProcessBench. We will revise the abstract to state that the improvements concern average downstream performance. On ProcessBench, RSP achieves 65.8 macro-average F1, compared with Qwen PRM's 70.5. ProcessBench focuses on identifying the first erroneous step, where fine-grained process annotations are particularly important. Qwen PRM uses additional large-scale process supervision, while RSP combines limited process annotations with outcome supervision. As discussed in Appendix B, process annotations remain important for this task. Our results demonstrate the benefit of this combination for average performance across reasoning search, response selection, and reinforcement learning.

We have also uploaded a revised version of the manuscript, in which all corresponding revisions are clearly marked at their respective locations for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.