acceptodds
Under review as a conference paper at ICLR 2027

Inducing Process Supervision from Outcome-Only Reinforcement Learning

Abstract

Process reward models (PRMs) have become a key component for LLMs, as their step-level feedback supports both post-training and test-time reasoning. However, training strong PRMs remains costly: human step annotation is difficult to scale, while Monte Carlo estimates is expensive and can drift from the intrinsic correctness of steps. To get effective PRMs at low cost, we introduce TIPS (Thinking-Induced Process Supervision), an outcome-only reinforcement learning (RL) framework for training generative PRMs. In TIPS, the model generates a chain-of-thought (CoT) followed by step-level labels and an outcome label. The reward depends solely on whether the predicted outcome matches the ground truth, and the resulting group-relative advantage is used to optimize the entire generated response. Intuitively, when checking intermediate steps helps determine the outcome, more accurate checks can lead to better outcome judgments and higher rewards. Outcome-only RL can therefore reinforce step-level verification without explicit process supervision. We validate the effectiveness of TIPS across math and agent benchmarks and four backbone families. Notably, TIPS-Qwen3-4B-Thinking-2507 reaches 85.2 F1 on ProcessBench with only 3.2K outcome-labeled trajectories, surpassing all evaluated trained PRMs and strong prompt-only judges such as GPT-5.4-Instruct and Claude-4.7-Opus, while still trailing o1-mini. Code and data are available at https://anonymous.4open.science/r/TIPS-01B5.

Primary area:
generative models
Keywords:
Large Language Models, Process Reward Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_twVrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TIPS, a framework for training generative process reward models (PRMs) using only trajectory-level outcome supervision. The model generates reasoning, step-level correctness labels, and a final outcome prediction. GRPO optimizes the entire response using rewards based solely on outcome prediction accuracy. The authors provide theoretical analysis relating outcome information to process information and evaluate the approach across mathematical reasoning and agent benchmarks. TIPS achieves 85.2 F1 on ProcessBench using a 4B model trained on 3.2K outcome-labeled trajectories, outperforming substantially larger PRMs.

Critical strengths
  • S1: TIPS-Qwen3-4B-Thinking improves ProcessBench F1 from 60.3 to 85.2 using only 3.2K outcome-labeled trajectories, outperforming several larger supervised PRMs. The matched-backbone comparisons with AutoPSV, ImplicitPRM, and SCAN-PRM also provide additional evidence that the approach is effective beyond simply using a strong pre-trained judge. Achieving this level of error localization without explicit process annotations is a meaningful contribution.
Critical weaknesses
  • W1: The ProcessBench results do not establish how much outcome-only RL improves error localization. ProcessBench F1 combines accuracy on erroneous and fully correct trajectories, but the paper reports only the combined score. This makes it difficult to separate improvements in identifying specific errors from improvements in recognizing correct solutions. More importantly, ProcessBench contains trajectories with correct final answers but incorrect intermediate reasoning. Since TIPS rewards only outcome prediction accuracy, it is unclear whether training improves verification in these cases, where outcome correctness provides no direct signal about the reasoning errors.

  • W2: Large improvements in process verification do not translate into comparable downstream gains. TIPS-Qwen3-4B-Thinking improves ProcessBench F1 from 60.3 to 85.2, but Best-of-8 accuracy improves only from 72.8 to 73.3. Moreover, the reranking procedure first filters candidates using outcome predictions, making it unclear how much the process scores contribute. The paper does not report this contribution separately or provide uncertainty estimates for the small downstream improvement.

Questions for authors
  • WQ1: Can the authors report ProcessBench accuracy separately on erroneous and fully correct trajectories, before and after TIPS training? In particular, how much of the improvement comes from identifying the exact first erroneous step?

  • WQ2: In Table 3, how does Best-of-8 reranking using only predicted outcome labels compare with the reported approach that also uses process scores? Are the improvements statistically significant?

5 comments
AuthorsOpenReview ↗

Additional Error Localization Results and Best-of-8 Ablations

We sincerely thank the reviewer for the valuable feedback. We address the comments and questions below.


WQ1 & W1: Error localization on erroneous trajectories, especially those with correct final answers.

Response:

We thank the reviewer for highlighting this distinction. We report accuracy before and after TIPS separately on erroneous and fully correct trajectories. Error Acc. measures exact first-error localization, while Correct Acc. measures recognition of error-free solutions.

ModelError Acc. ↑Correct Acc. ↑
Qwen3-4B-Instruct-2507†59.768.7
+ TIPS75.2 (+15.4)80.5 (+11.8)
Qwen3-4B-Thinking-250776.649.3
+ TIPS80.5 (+3.9)88.8 (+39.6)
DS-R1-Distill-Qwen2.5-7B35.265.0
+ TIPS47.5 (+12.3)84.7 (+19.7)
DS-R1-Distill-Llama-8B32.246.8
+ TIPS55.5 (+23.3)73.9 (+27.1)
Qwen3-8B-Thinking59.361.4
+ TIPS71.1 (+11.8)87.2 (+25.9)

TIPS improves both exact first-error localization and recognition of correct solutions across all five backbones. We further evaluate the 521 ProcessBench trajectories with incorrect intermediate steps but correct final answers:

ModelFirst-error localization accuracy (%)
Qwen3-4B-Instruct-250758.3
+ TIPS71.4 (+13.1 pp)
Qwen3-4B-Thinking-250767.8
+ TIPS72.2 (+4.4 pp)

As shown in the table, TIPS improves first-error localization by 13.1 and 4.4 pp on this subset, showing that its localization gains also extend to cases where the final answer is correct despite intermediate errors.

WQ2 & W2: Best-of-8 reranking using only predicted outcome labels.

Response:

Thank you for raising this question. The table below reports Best-of-8 accuracy under three scoring rules on the same candidate sets. PRM-only selects the candidate with the highest proportion of steps predicted to be correct. ORM-only selects the first candidate whose final answer is predicted to be correct. ORM → PRM is the two-stage rule used in the paper. It filters candidates by predicted outcome and ranks the retained candidates by their mean process score. If no candidate passes the filter, it falls back to PRM-only.

Judge modelPRM-onlyORM-onlyORM → PRM
Qwen3-4B-Thinking-250770.673.072.8
TIPS-Qwen3-4B-Thinking-250772.073.073.3
TIPS − Base (pp)+1.40.0+0.5
Qwen3-4B-Instruct-250770.171.671.6
TIPS-Qwen3-4B-Instruct-250770.872.672.8
TIPS − Base (pp)+0.7+1.0+1.2

As shown in the table, ORM-only is already a strong baseline for selecting correct answers, and the additional gains from process ranking are not statistically significant. For the base Thinking and Instruct models, adding process ranking changes accuracy by −0.2 and 0.0 pp, respectively; after TIPS training, the corresponding gains become +0.3 and +0.2 pp. Notably, TIPS improves PRM-only accuracy by 1.4 and 0.7 pp on the two backbones, respectively, with both gains being statistically significant. The corresponding 95% confidence intervals from stratified paired bootstrap resampling are [0.7, 2.0] and [0.1, 1.3], both excluding zero. These results support our main claim that outcome-only RL can induce process verification.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer twVr,

Thank you again for your helpful suggestions. We have added error-localization results, including cases with correct final answers but incorrect reasoning, and Best-of-8 scoring ablations with significance analysis. With only three days remaining in the discussion period, we would appreciate any follow-up feedback and would be happy to clarify or discuss any remaining questions.

AuthorsOpenReview ↗

Follow-up on the Additional Experiments Before Tomorrow’s Deadline

Dear Reviewer twVr,

With the discussion ending tomorrow, we wanted to follow up on two points you raised. Our rebuttal now reports how well TIPS locates reasoning errors even when the final answer is correct. We also compared Best-of-8 scoring rules and tested the statistical significance of the results. We’d appreciate hearing whether these analyses address your concerns and whether you’d consider revisiting your overall assessment.

Thank you again for your time and feedback.

Reviewer_5LsROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TIPS, a method for training process reward models using only outcome labels. The model generates a reasoning trace, step-level judgments, and an outcome prediction, while reinforcement learning rewards only outcome accuracy. Experiments on mathematical and agent reasoning show that this can substantially improve process verification, particularly for models with strong chain-of-thought capabilities.

Critical strengths

The paper’s main strength is its strong data efficiency, TIPS learns effective process verification from only 3.2K outcome-labeled trajectories and outperforms several process reward models trained with substantially more supervision. The chain-of-thought versus label-only ablation is also compelling, as it shows that the improvement depends on explicit reasoning rather than outcome supervision alone. These results support both the practical value of the method and its proposed mechanism.

Critical weaknesses

The main weakness is a gap between the theoretical analysis and the broader claim that outcome supervision induces reliable process supervision. The theory establishes that the generated reasoning may contain information about step correctness, but it does not show that the model’s explicit step labels accurately or faithfully recover this information. A model could potentially learn outcome-correlated heuristics while still producing plausible but unreliable process judgments. This connection should either be established empirically or reflected more cautiously in the paper’s claims.

The evaluation also leaves some uncertainty about robustness and generalization. In particular, the paper does not report a mathematical decontamination analysis comparable to the one provided for the agent-domain experiments, which is important given that the training trajectories are derived from mathematical datasets related to the evaluation setting. Moreover, the downstream Best-of-8 improvements are relatively modest compared with the large ProcessBench gains, and the agent-domain improvements are not consistent across all backbones and tasks. Reporting results across multiple training and inference seeds, along with stronger contamination checks and scoring-rule ablations, would help determine whether the reported process-verification gains are robust and translate reliably into practical response selection.

Questions for authors
  1. The theoretical analysis suggests that the generated reasoning contains information about step correctness. Is there a way to show more directly that this information is actually reflected in the model’s predicted step labels, rather than only helping with the final outcome prediction?

  2. In the Best-of-8 evaluation, how much of the improvement comes from the outcome score versus the step-level process score? Results using each score separately would make the practical contribution of the learned process labels clearer.

  3. Have the authors considered a matched supervised or rejection-sampling baseline? Such a comparison would help clarify whether reinforcement learning is essential or whether similar gains could be obtained through a simpler training procedure.

5 comments
AuthorsOpenReview ↗

Evidence for Process Verification, Best-of-8 Ablations, and the Role of Reinforcement Learning

We sincerely thank the reviewer for the valuable feedback. We address the comments and questions below.


W1 & Q1: Is the step-correctness information in the generated reasoning reflected in the predicted step labels?

Response:

We thank the reviewer for raising this question. We provide two complementary empirical checks:

  1. Accuracy of the predicted step labels. On ProcessBench and AgentProcessBench, the predicted first-error position is derived directly from the generated step labels and evaluated against expert annotations. The improvements in Tables 1 and 2 therefore demonstrate more accurate process judgments in the explicit labels, rather than merely better outcome predictions. The erroneous-sample breakdown further shows that first-error localization accuracy improves by 15.4 pp for Qwen3-4B-Instruct-2507 and 3.9 pp for Qwen3-4B-Thinking-2507.

  2. Agreement with the generated thought. In Appendix G, human annotators evaluated 60 ProcessBench trajectories. TIPS-Qwen3-4B-Instruct-2507 correctly identified the first erroneous step or recognized an error-free solution in 47 cases. In 38 of these 47 cases (80.9%; 63.3% of all 60), the thought also provided a correct explanation for the assessment. This supports agreement between the generated reasoning and the explicit process judgments.

We agree that the theory does not explicitly guarantee accurate step labels. We will clarify this limitation and distinguish the theoretical information bound from the empirical evidence.

Q2: How much do outcome and process scores contribute to Best-of-8 reranking?

Response:

Thank you for raising this question. The table below reports Best-of-8 accuracy under three scoring rules on the same candidate sets. PRM-only selects the candidate with the highest proportion of steps predicted to be correct. ORM-only selects the first candidate whose final answer is predicted to be correct. ORM → PRM is the two-stage rule used in the paper. It filters candidates by predicted outcome and ranks the retained candidates by their mean process score. If no candidate passes the filter, it falls back to PRM-only.

Judge modelPRM-onlyORM-onlyORM → PRM
Qwen3-4B-Thinking-250770.673.072.8
TIPS-Qwen3-4B-Thinking-250772.073.073.3
TIPS − Base (pp)+1.40.0+0.5
Qwen3-4B-Instruct-250770.171.671.6
TIPS-Qwen3-4B-Instruct-250770.872.672.8
TIPS − Base (pp)+0.7+1.0+1.2

From the table, we find that:

  1. TIPS improves PRM-only reranking on both backbones. PRM-only accuracy increases by 1.4 pp for Thinking and 0.7 pp for Instruct, showing that the learned step labels improve response selection even without outcome scores.

  2. Process ranking contributes more after TIPS training. Adding process ranking after outcome filtering changes Best-of-8 accuracy by −0.2 pp for the base Thinking model and 0.0 pp for the base Instruct model. After TIPS training, it improves accuracy by +0.3 pp and +0.2 pp, respectively.

Q3: Is reinforcement learning necessary, or can matched supervised or rejection-sampling training achieve similar gains?

Response:

We thank the reviewer for suggesting this comparison. Starting from the same Qwen3-4B-Instruct-2507 checkpoint, we compare rejection-sampling SFT with TIPS using the same 3.2K samples and same rollout quota (8 for each question). For rejection-sampling SFT, we retain outputs whose predicted outcome labels match the ground truth and apply SFT to the retained outputs. The table reports ProcessBench F1, with gains over the base model.

MethodGSM8KMATHOlympiadBenchOmniMathAvg.
Qwen3-4B-Instruct-250764.868.763.658.163.8
Rejection-sampling SFT73.4 (+8.6)77.6 (+8.9)69.6 (+6.0)66.5 (+8.4)71.7 (+7.9)
TIPS (RL)84.6 (+19.8)82.6 (+13.9)74.5 (+10.9)69.0 (+10.9)77.7 (+13.9)

From the table, we find that:

  1. Rejection-sampling SFT also improves process verification. It increases average F1 by 7.9 pp, showing that RL is not necessary to obtain some improvement.

  2. TIPS yields larger gains in this comparison. It improves average F1 by 13.9 points, exceeding rejection-sampling SFT by 6.0 points and achieving higher F1 on all four subsets.

AuthorsOpenReview ↗

Robustness Across Seeds and Mathematical Decontamination Analysis

W2(a): Results of different seeds.

Response:

We thank the reviewer for these suggestions. We report results from three independent training seeds for TIPS models and three inference seeds for the untrained base models, with all scores presented as mean ± SD. The ProcessBench F1 scores (mean ± SD) are reported below:

ModelGSM8KMATHOlympiadBenchOmniMathAvg.
TIPS-Qwen3-4B-Instruct-250784.2 ± 1.082.8 ± 0.574.7 ± 0.771.2 ± 2.178.2 ± 0.5
TIPS-Qwen3-4B-Thinking-250789.0 ± 0.587.0 ± 0.684.1 ± 0.779.0 ± 0.984.8 ± 0.4

The average F1 exhibits low variance across three training seeds, with standard deviations of 0.5 and 0.4, respectively.

The AgentProcessBench FirstErrAcc scores (mean ± SD) are reported below:

ModelHotPotQAGAIABFCLτ₂-BenchAvg.
Qwen3-4B-Instruct-250755.2 ± 1.431.7 ± 0.832.8 ± 0.843.2 ± 0.040.7 ± 0.6
+ TIPS58.3 ± 1.239.7 ± 1.333.7 ± 0.548.8 ± 1.145.1 ± 0.4
Qwen3-4B-Thinking-250759.5 ± 0.232.3 ± 1.635.3 ± 0.646.5 ± 1.443.4 ± 0.5
+ TIPS59.3 ± 0.838.9 ± 1.942.4 ± 2.853.2 ± 0.848.5 ± 1.0

These results show that TIPS achieves higher average FirstErrAcc than the corresponding base models across seeds, with mean gains of 4.4 and 5.1 points for the Instruct and Thinking backbones, respectively.

W2(b): Mathematical decontamination analysis.

Response:

We conducted an overlap audit between the mathematical training trajectories and ProcessBench, using the same 5-gram Jaccard similarity threshold of 0.8 as in our agent-domain decontamination analysis, with additional normalization of mathematical formatting. We identified 4 of the 3,200 training trajectories (0.125%) whose questions exceeded this threshold against at least one ProcessBench problem. After removing these four trajectories, we reran TIPS RL training on Qwen3-4B-Instruct-2507 and obtained a ProcessBench F1 of 77.9, compared with the originally 78.2 ± 0.5. This indicates that the detected overlap does not account for the reported ProcessBench gains.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer 5LsR,

Thank you again for your constructive feedback. We have added error-localization results, scoring-rule ablations, a matched rejection-sampling SFT baseline, multi-seed results, and a mathematical decontamination experiment. With only three days remaining in the discussion period, we would appreciate your thoughts on these additions and the opportunity to address any remaining concerns.

Reviewer_zvMkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TIPS, which trains a generative PRM with outcome-only RL. The model writes a chain of thought, then step labels and an outcome label, and GRPO rewards only the outcome label. An information-theoretic argument (Fano's inequality plus the chain rule) lower-bounds how much the thought must carry about step correctness. With 3.2K outcome-labeled MATH trajectories, TIPS-Qwen3-4B-Thinking reaches 85.2 F1 on ProcessBench, above all listed trained PRMs and GPT-5.4 / Claude-4.7-Opus judges. It also improves AgentProcessBench, Best-of-8 reranking and six other backbones.

Critical strengths
  • A simple, cheap recipe with strong results. Outcome labels only, 3.2K trajectories and a 4B model give +24.9 F1 over the same backbone used as a judge (60.3 → 85.2) and beat matched AutoPSV and ImplicitPRM trained on 197K trajectories by over 30 F1.

  • Useful analysis beyond the headline. The paper shows the gain needs a thought (label-only prompting: +0.7 F1 vs. +13.9), is not explained by better problem solving (Appendix H, solver-RLVR control), and reports a failure mode where Qwen2.5-7B-Instruct's process F1 degrades while ORM accuracy keeps rising (Fig. 6).

Critical weaknesses
  • Part of the gain may be outcome judging, not step localization. ProcessBench F1 is the harmonic mean of accuracy on erroneous and on fully correct solutions, and on correct solutions the task reduces to an outcome judgment, which is exactly what TIPS is rewarded for. The paper does not report the two accuracies separately, so how much of the +14 to +25 F1 comes from better first-error localization is unclear. Similarly, in Best-of-8 TIPS adds only +0.5 / +1.2 points over its own untrained backbone (72.8 → 73.3, 71.6 → 72.8), and the agent gains are small and mixed (e.g. −3.2 on HotPotQA for Qwen3-8B). Table 2 has no PRM baselines other than the base models.

  • The theory does not explain localization, and a key baseline is missing. The right-hand side of Eq. 5 is at most H(D | R) ≤ 1 bit, so the bound holds if the thought only encodes "some step is wrong", without saying which step. It therefore cannot account for first-error accuracy, which is what ProcessBench measures. On the empirical side, there is no same-backbone generative PRM trained with process labels (e.g. step-level reward or SFT using the SCAN labels of the same 3.2K trajectories). That comparison would show how much is lost by dropping process labels, which is the paper's central trade-off. Results also appear to come from a single run, with no variance reported.

Questions for authors

Clarifications:

  1. Can you report ProcessBench accuracy on erroneous and on correct samples separately, before and after TIPS?
  2. How was the final checkpoint chosen for each model? Was ProcessBench used, given that Fig. 6 shows process F1 can peak early? Fig. 4 reports 10.7 → 19.0 for Qwen2.5-7B-Instruct, but Fig. 6 gives a final ΔPRM of +9.4; which is correct?
  3. How often are the step labels inconsistent with the outcome label (e.g. all steps 1 but outcome 0)? Did the 60 human-evaluated samples in Appendix G include correct solutions?

Experiments that could change my assessment:

  1. A same-backbone generative PRM trained with process labels on the same 3.2K trajectories.
  2. Results over several seeds, and AgentProcessBench baselines beyond the base models.
6 comments
AuthorsOpenReview ↗

Error Localization, Checkpoint Selection, and Step–Outcome Label Consistency

We sincerely thank the reviewer for the valuable feedback. We address the comments and questions below.


Q1: ProcessBench accuracy on erroneous and on correct samples.

Response:

We thank the reviewer for this suggestion. The breakdown below shows that TIPS improves both first-error localization on erroneous solutions (+3.9 to +23.3 pp) and recognition of correct solutions (+11.8 to +39.6 pp) across all four backbones.

ModelError Acc. ↑Correct Acc. ↑
Qwen3-4B-Instruct-2507†59.768.7
+ TIPS75.2 (+15.4)80.5 (+11.8)
Qwen3-4B-Thinking-250776.649.3
+ TIPS80.5 (+3.9)88.8 (+39.6)
DS-R1-Distill-Qwen2.5-7B35.265.0
+ TIPS47.5 (+12.3)84.7 (+19.7)
DS-R1-Distill-Llama-8B32.246.8
+ TIPS55.5 (+23.3)73.9 (+27.1)

Q2: Checkpoint selection procedure.

Response:

We use the final checkpoint from each training run, without using ProcessBench for checkpoint selection. For example, for Qwen2.5-7B-Instruct, we do not report the peak ProcessBench gain of +25.7 F1 in Fig. 4. The discrepancy between Fig. 4 (+8.3) and Fig. 6 (+9.4) arose from inadvertently using results from different runs. The correct gain is +8.3 F1, as reported in Fig. 4. We apologize for this inconsistency and will correct Fig. 6 and explicitly state the checkpoint selection procedure in the revised paper.

Q3: Step–Outcome Label Consistency and Appendix G Sample Composition.

Response:

On ProcessBench, TIPS-Qwen3-4B-Instruct-2507 and TIPS-Qwen3-4B-Thinking-2507 both output all step labels as 1 but an outcome label of 0 in approximately 0.1% of samples. In contrast, they output at least one step label of 0 with an outcome label of 1 in 11.2% and 13.4% of samples, respectively.

Note that a solution can contain incorrect intermediate steps yet reach a correct final answer, so predicting a step label of 0 with an outcome label of 1 is not necessarily inconsistent. In the ground-truth annotations, 15.3% of samples contain incorrect steps but have correct final answers. And no samples have all steps correct but an incorrect final answer. On these 521 samples, TIPS improves first-error localization accuracy by 13.1 and 4.4 pp, respectively:

ModelError Acc. (%)
Qwen3-4B-Instruct-250758.3
+ TIPS71.4 (+13.1 pp)
Qwen3-4B-Thinking-250767.8
+ TIPS72.2 (+4.4 pp)

These results show that TIPS improves error localization even when incorrect intermediate steps lead to a correct final answer.

The 60 human-evaluated samples in Appendix G include 31 erroneous solutions and 29 correct solutions.

AuthorsOpenReview ↗

Process-Supervised Comparison, Multi-Seed Results, and More AgentProcessBench Baselines

Q4: A same-backbone generative PRM trained with process labels on the same 3.2K trajectories.

Response:

We thank the reviewer for suggesting this important baseline. Following your suggestion, we added a process-supervised RL baseline using the same Qwen3-4B-Instruct-2507 backbone and the same 3.2K trajectories. The reward is the fraction of predicted step labels matching the SCAN step annotations, with all other hyperparameters unchanged.

The ProcessBench F1 scores are shown below:

ModelGSM8KMATHOlympiadBenchOmniMathAvg.
Qwen3-4B-Instruct-250764.868.763.658.163.8
+ Outcome RL (TIPS)84.6 (+19.8)82.6 (+13.9)74.5 (+10.9)69.0 (+10.9)77.7 (+13.9)
+ Process RL85.2 (+20.4)83.7 (+15.0)77.7 (+14.1)72.7 (+14.6)79.8 (+16.0)

Using SCAN’s denoised Monte Carlo step labels provides stronger supervision and improves average F1 by 2.1 points over outcome-only RL. However, TIPS retains approximately 87% of the improvement over the base model (+13.9 vs. +16.0), without requiring expensive Monte Carlo rollouts for step-level annotation.

Q5: Results over several seeds, and AgentProcessBench baselines beyond the base models.

Response:

We thank the reviewer for these suggestions. We report results from three independent training seeds for TIPS models and three inference seeds for the untrained base models, with all scores presented as mean ± SD.

The ProcessBench F1 scores (mean ± SD) are reported below:

ModelGSM8KMATHOlympiadBenchOmniMathAvg.
TIPS-Qwen3-4B-Instruct-250784.2 ± 1.082.8 ± 0.574.7 ± 0.771.2 ± 2.178.2 ± 0.5
TIPS-Qwen3-4B-Thinking-250789.0 ± 0.587.0 ± 0.684.1 ± 0.779.0 ± 0.984.8 ± 0.4

The average F1 exhibits low variance across three training seeds, with standard deviations of 0.5 and 0.4, respectively.


We also added the following AgentProcessBench baselines:

  1. Frontier foundation models: GPT-6-Luna, Gemini-4-Argon, and DeepSeek-V4.1-Flash-1017.
  2. Agent process reward model: the publicly available ToolPRM-GRPO [1].
  3. Larger model and distillation baseline: Qwen3-30B-A3B-Instruct-2507 and a same-backbone SFT baseline distilled from it using the same training samples as TIPS.

We report FirstErrAcc as mean ± SD over three independent runs:

Frontier models

ModelHotPotQAGAIABFCLτ₂-BenchAvg.
GPT-6-Luna68.8 ± 1.262.5 ± 1.858.0 ± 2.443.3 ± 2.858.2 ± 0.9
Gemini-4-Argon73.6 ± 0.866.4 ± 0.464.4 ± 0.862.3 ± 0.866.7 ± 0.6
DeepSeek-V4.1-Flash-101773.1 ± 1.662.1 ± 2.657.5 ± 3.356.7 ± 1.062.3 ± 1.6

Baselines and TIPS

ModelHotPotQAGAIABFCLτ₂-BenchAvg.
ToolPRM-GRPO57.1 ± 2.037.3 ± 2.032.3 ± 1.445.1 ± 3.242.9 ± 2.0
Qwen3-30B-A3B-Instruct-250756.8 ± 0.832.3 ± 2.334.7 ± 1.249.6 ± 0.443.3 ± 0.3
Qwen3-4B-Instruct-250755.2 ± 1.431.7 ± 0.832.8 ± 0.843.2 ± 0.040.7 ± 0.6
+ SFT (from Qwen3-30B-A3B-Instruct-2507)56.3 ± 1.331.6 ± 1.435.3 ± 0.947.6 ± 1.442.7 ± 0.5
+ TIPS58.3 ± 1.239.7 ± 1.333.7 ± 0.548.8 ± 1.145.1 ± 0.4
Qwen3-4B-Thinking-250759.5 ± 0.232.3 ± 1.635.3 ± 0.646.5 ± 1.443.4 ± 0.5
+ TIPS59.3 ± 0.838.9 ± 1.942.4 ± 2.853.2 ± 0.848.5 ± 1.0

The results show:

  1. Frontier models retain the lead. All three frontier models achieve higher average FirstErrAcc than TIPS on AgentProcessBench, indicating that a performance gap remains.
  2. Comparison with additional baselines: TIPS-Qwen3-4B-Instruct achieves an average score of 45.1, outperforming ToolPRM-GRPO (42.9), Qwen3-30B-A3B-Instruct-2507 (43.3), and the matched-data SFT baseline (42.7).
  3. Improvements over base models: TIPS improves the average scores of the Instruct and Thinking backbones by 4.4 and 5.1 points, respectively, although gains vary across tasks.

References:

[1] Li, Dawei, et al. "Toolprmbench: Evaluating and advancing process reward models for tool-using agents." Findings of the Association for Computational Linguistics: ACL 2026.

AuthorsOpenReview ↗

Best-of-8 and AgentProcessBench Gains, and the Scope of the Theoretical Analysis

W1: The gains on Best-of-8 and AgentProcessBench are relatively small.

Response:

We agree that the Best-of-8 gains are modest. Our main claim is that outcome-only training can improve process verification; Best-of-8 serves as a supplementary evaluation of downstream answer selection. Q1 directly demonstrates improved first-error localization on ProcessBench. On AgentProcessBench, the additional results in Q5 show that TIPS-Qwen3-4B-Instruct-2507 achieves an average FirstErrAcc of 45.1, outperforming Qwen3-30B-A3B-Instruct-2507 by 1.8 pp and the same-backbone SFT model distilled from it using the same training samples by 2.4 pp, although gains vary across tasks.

W2: The gap between theory and experiments.

Response:

We agree that the lower bound in Eq. 5 does not guarantee first-error localization. We clarify two relevant empirical findings:

  1. Process verification in generated thoughts. In Appendix G, human annotators found that TIPS-Qwen3-4B-Instruct-2507 provided correct explanations in 38 of the 47 correctly assessed cases (80.9%), supporting meaningful process verification in its thoughts.

  2. Dependence on model priors. Section 4.4 shows that improved outcome prediction does not always lead to improved error localization. Qwen2.5-7B-Instruct exhibits degraded process F1 despite increasing outcome accuracy, whereas its DeepSeek-R1-distilled counterpart benefits from TIPS. These findings suggest that reasoning priors affect whether outcome-only training improves localization.

We will narrow our interpretation of Eq. 5 and clarify that localization gains are supported by empirical evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.