- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Reasoning State Propagation (RSP) which uses outcome labels to guide the training of PRMs. The authors first introduce the repair probability to capture the state when the LLMs resolve earlier errors. And they define a state propagation matrix which allows the outcome supervision to guide the process states. Then, PRM can be trained with less reliance on costly process annotations. Experiment results shown that the proposed RSP outperforms other compared methods on different datasets and LLMs.
Critical strengths
1.The motivation is clear and valuable.RSP explores how to leverage low-cost scalable outcome supervision during PRM training.
2.This paper introduce a new repair probability to capture the self-correction step which is reasonable for current reasoning LLMs.
3.The experiments cover multiple test-time scaling methods and reinforcement learning to show the effectiveness of the proposed RSP.
4.RSP outperforms most PRMs on different datasets and LLMs.
Critical weaknesses
1.Table.1 lacks the results compared with Qwen2.5-Math-PRM. Authors should add the experiments such as Qwen2.5-Math-7B with RSP or Qwen3 with Qwen2.5-Math-PRM.
2.Although the state propagation matrix allows the final state to be computed from previous state transitions, Eq.5 can not identifies which steps introduce or correct errors. A correct final answer can cause the model to decrease all break probabilities and increase all repair probabilities.
3.RSP uses PRM800K as the process-annotated training set but it contains no B->G labels. How does RSP learns to identify repair steps?
Questions for authors
See Weaknesses
3 comments
Weakness #3: Learning repair without B→G labels in PRM800K
PRM800K primarily supervises states before and at the first error. Repair is therefore learned indirectly from complete outcome-annotated trajectories, using the same model trained on process labels. If an earlier prefix is predicted invalid but the final outcome is correct, fitting both predictions requires the remaining transitions to allow recovery. Appendix C.3 formalizes this: when and , Here, is the probability of reaching a valid terminal state from B through the remaining steps. Conditional on preserving the invalid prefix prediction, the positive outcome requires sufficient recovery in those steps. This explains how repair receives supervision without explicit B→G labels. The terminal supervision constrains the cumulative probability of recovery over the remaining trajectory, while the transition predictors parameterize how this constraint is distributed across subsequent reasoning steps. Whether these transitions align with actual recovery events is further examined in our recovery evaluation below.
To evaluate that behavior, we use GPT-6.1 Sol to annotate Qwen3-1.7B solutions to DAPO-Math-17K questions. Labels indicate whether the current prefix contains an unresolved error, including after the first error. A locally correct new step is not a repair if the earlier error still affects the current derivation. We evaluate 19,111 adjacent pairs with a preceding B state, including 3,376 B→G and 15,735 B→B pairs. Both models score the same pairs without retraining. F1 uses a threshold of 0.5.
| Method | AUROC | AUPRC | F1 |
|---|---|---|---|
| Qwen2.5-Math-PRM-7B | 0.7053 | 0.2659 | 0.4086 |
| RSP | 0.7788 | 0.3794 | 0.4616 |
RSP performs better on all three metrics, supporting its ability to distinguish recovery from error persistence under these annotations.
We thank the reviewer for this suggestion. It also inspires us to further improve RSP by incorporating directly annotated B→G examples into training. In fact, collecting such data is feasible with strong models such as GPT-6. We believe that such data could further improve RSP by providing direct supervision for repair transitions. We plan to explore this in future work.
We have also uploaded a revised version of the manuscript, in which all corresponding revisions are clearly marked at their respective locations for ease of reference.
Dear reviewer:
Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.
Sincerely,
Authors
Dear Reviewer R9a6,
Thank you for recognizing the motivation, repair formulation, and downstream evaluation. We clarify the comparison in Table 1 and explain how repair is trained and evaluated.
Weakness #1: Comparison with Qwen2.5-Math-PRM in Table 1
Following the reviewer’s suggestion, we additionally train RSP using Qwen2.5-Math-7B as the backbone and compare it with Qwen2.5-Math-PRM-7B. The results are shown below.
RSP achieves higher accuracy in both settings, improving beam search by 4.6 percentage points and BoN by 0.4 percentage points over Qwen2.5-Math-PRM-7B. The alternative comparison using Qwen3 with Qwen2.5-Math-PRM cannot be conducted under matched training conditions, as the training data used for Qwen2.5-Math-PRM have not been publicly released. We will include the above results in the revised version.
Weakness #2: Whether a correct outcome simply lowers all break scores and raises all repair scores
Eq. (5) defines state propagation, while the learning objectives in Eqs. (8) and (9) train the transitions using process and outcome labels. Process labels constrain intermediate validity, and outcome labels train preceding transitions through the final state. This joint training is essential to interpreting the learned break and repair scores.
Appendix C.3 gives the outcome gradients with respect to the break and repair logits:
=\frac{p_{t-1}^G\alpha_t(1-\alpha_t)}{Z_{\mathrm{out}}} \bigl(h_t(G)-h_t(B)\bigr),$$ $$\frac{\partial\ell_{\mathrm{out}}}{\partial\eta_t^{\mathrm{rp}}} =-\frac{p_{t-1}^B\beta_t(1-\beta_t)}{Z_{\mathrm{out}}} \bigl(h_t(G)-h_t(B)\bigr).$$ Here, $h_t(a)$ is the probability of reaching the observed terminal target from state $a$ through the remaining steps, and $Z_{\mathrm{out}}$ is the probability of that target. The gradients depend on the preceding state and the subsequent transitions. In particular, the repair gradient is weighted by $p_{t-1}^B$, and both gradients vanish when the remaining transitions make the terminal target equally likely from G and B. When $h_t(G)>h_t(B)$, the outcome loss favors lower break and higher repair at that position. These directions can agree across steps, but the update magnitudes depend on the preceding states and remaining transitions. Process labels also constrain intermediate validity, including invalid states, so predicting every prefix as valid would conflict with those annotations. We further evaluate whether RSP can distinguish error recovery from error persistence in our response to Weakness #3 you proposed, so you can find more details below.