- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TIPS, a framework for training generative process reward models (PRMs) using only trajectory-level outcome supervision. The model generates reasoning, step-level correctness labels, and a final outcome prediction. GRPO optimizes the entire response using rewards based solely on outcome prediction accuracy. The authors provide theoretical analysis relating outcome information to process information and evaluate the approach across mathematical reasoning and agent benchmarks. TIPS achieves 85.2 F1 on ProcessBench using a 4B model trained on 3.2K outcome-labeled trajectories, outperforming substantially larger PRMs.
Critical strengths
- S1: TIPS-Qwen3-4B-Thinking improves ProcessBench F1 from 60.3 to 85.2 using only 3.2K outcome-labeled trajectories, outperforming several larger supervised PRMs. The matched-backbone comparisons with AutoPSV, ImplicitPRM, and SCAN-PRM also provide additional evidence that the approach is effective beyond simply using a strong pre-trained judge. Achieving this level of error localization without explicit process annotations is a meaningful contribution.
Critical weaknesses
-
W1: The ProcessBench results do not establish how much outcome-only RL improves error localization. ProcessBench F1 combines accuracy on erroneous and fully correct trajectories, but the paper reports only the combined score. This makes it difficult to separate improvements in identifying specific errors from improvements in recognizing correct solutions. More importantly, ProcessBench contains trajectories with correct final answers but incorrect intermediate reasoning. Since TIPS rewards only outcome prediction accuracy, it is unclear whether training improves verification in these cases, where outcome correctness provides no direct signal about the reasoning errors.
-
W2: Large improvements in process verification do not translate into comparable downstream gains. TIPS-Qwen3-4B-Thinking improves ProcessBench F1 from 60.3 to 85.2, but Best-of-8 accuracy improves only from 72.8 to 73.3. Moreover, the reranking procedure first filters candidates using outcome predictions, making it unclear how much the process scores contribute. The paper does not report this contribution separately or provide uncertainty estimates for the small downstream improvement.
Questions for authors
-
WQ1: Can the authors report ProcessBench accuracy separately on erroneous and fully correct trajectories, before and after TIPS training? In particular, how much of the improvement comes from identifying the exact first erroneous step?
-
WQ2: In Table 3, how does Best-of-8 reranking using only predicted outcome labels compare with the reported approach that also uses process scores? Are the improvements statistically significant?
5 comments
A Gentle Follow-up on Our Rebuttal
Dear Reviewer twVr,
Thank you again for your helpful suggestions. We have added error-localization results, including cases with correct final answers but incorrect reasoning, and Best-of-8 scoring ablations with significance analysis. With only three days remaining in the discussion period, we would appreciate any follow-up feedback and would be happy to clarify or discuss any remaining questions.
Follow-up on the Additional Experiments Before Tomorrow’s Deadline
Dear Reviewer twVr,
With the discussion ending tomorrow, we wanted to follow up on two points you raised. Our rebuttal now reports how well TIPS locates reasoning errors even when the final answer is correct. We also compared Best-of-8 scoring rules and tested the statistical significance of the results. We’d appreciate hearing whether these analyses address your concerns and whether you’d consider revisiting your overall assessment.
Thank you again for your time and feedback.
Additional Error Localization Results and Best-of-8 Ablations
We sincerely thank the reviewer for the valuable feedback. We address the comments and questions below.
WQ1 & W1: Error localization on erroneous trajectories, especially those with correct final answers.
Response:
We thank the reviewer for highlighting this distinction. We report accuracy before and after TIPS separately on erroneous and fully correct trajectories. Error Acc. measures exact first-error localization, while Correct Acc. measures recognition of error-free solutions.
TIPS improves both exact first-error localization and recognition of correct solutions across all five backbones. We further evaluate the 521 ProcessBench trajectories with incorrect intermediate steps but correct final answers:
As shown in the table, TIPS improves first-error localization by 13.1 and 4.4 pp on this subset, showing that its localization gains also extend to cases where the final answer is correct despite intermediate errors.
WQ2 & W2: Best-of-8 reranking using only predicted outcome labels.
Response:
Thank you for raising this question. The table below reports Best-of-8 accuracy under three scoring rules on the same candidate sets. PRM-only selects the candidate with the highest proportion of steps predicted to be correct. ORM-only selects the first candidate whose final answer is predicted to be correct. ORM → PRM is the two-stage rule used in the paper. It filters candidates by predicted outcome and ranks the retained candidates by their mean process score. If no candidate passes the filter, it falls back to PRM-only.
As shown in the table, ORM-only is already a strong baseline for selecting correct answers, and the additional gains from process ranking are not statistically significant. For the base Thinking and Instruct models, adding process ranking changes accuracy by −0.2 and 0.0 pp, respectively; after TIPS training, the corresponding gains become +0.3 and +0.2 pp. Notably, TIPS improves PRM-only accuracy by 1.4 and 0.7 pp on the two backbones, respectively, with both gains being statistically significant. The corresponding 95% confidence intervals from stratified paired bootstrap resampling are [0.7, 2.0] and [0.1, 1.3], both excluding zero. These results support our main claim that outcome-only RL can induce process verification.