acceptodds
Under review as a conference paper at ICLR 2027

DeltaPRM: Rollout-Free Automatic Process Rewards via Answer Likelihood Gain for Retrieval Agents

Abstract

Retrieval-augmented language agents have become a central paradigm for knowledge-intensive question answering, yet they are typically trained with outcome rewards alone, leaving individual search interactions unsupervised. Process reward models (PRMs) address this gap, but their step-level labels are expensive to obtain: human annotation and LLM-as-a-judge labels both demand substantial resources, while Monte-Carlo (MC) estimation, the dominant annotation-free alternative, becomes prohibitive for retrieval agents, since each per-step continuation triggers additional interaction with the search environment. We introduce DeltaPRM, a process reward that requires neither extra rollouts nor extra environment interaction. DeltaPRM scores each retrieval call and the reasoning that follows it by how much the information it brings raises the model's likelihood of the correct final answer — the delta between steps. Labeling thus reduces to a single forward pass per step and relies only on outcome supervision already available in any QA dataset. We train a PRM to predict the discounted return of this label signal from the reasoning prefix alone, no longer requiring access to the ground truth final answer. Across multi-hop agentic search benchmarks, we test DeltaPRM on best-of- answer selection, step-level guided generation (greedy decoding, beam search, and MC tree search), step-level verification on AgentProcessBench, and online RL, where it consistently matches or surpasses substantially more expensive baselines, including LLM-as-a-judge PRMs. At the same time, DeltaPRM reduces labeling cost by and relative to MC estimation and LLM-as-a-judge, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward model, automatic process reward model, agentic search, test time scaling, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NaRvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DeltaPRM, a process reward model for retrieval agents trained using changes in the frozen generator's teacher-forced log-likelihood of a reference answer between successive reasoning/search steps. Discounted sums of these changes are converted to soft labels, and the PRM is trained jointly against these labels and trajectory-level outcome labels. Label construction requires no additional sampled continuations, retrieval calls, or external judge, although the training trajectories themselves must first be generated. After training, the PRM can score prefixes without reference answers. Experiments cover best-of-N selection, guided decoding, human-labeled step verification, and GRPO, with additional generator and domain transfer. The strongest evidence is on multi-hop search; results outside this setting are often comparable to established PRMs rather than uniformly superior.

Critical strengths

A practical supervision mechanism that avoids additional environment interaction. Reference-answer likelihood differences provide a graded, signed signal from supervision already present in QA data. Distilling that signal into a prefix scorer removes the need for reference answers at deployment. This is a consequential construction for retrieval agents, where continuation-based labeling entails expensive search interaction. Substantial evidence of useful guidance on multi-hop search. On HotpotQA, DeltaPRM reaches 60.8% under beam search and 63.2% under MCTS, versus 57.0% and 59.6% for the matched-backbone ORM (Table 1). The prefix-broadcast outcome control is also weaker (Tables 5 and 25). The paper evaluates multiple datasets and generator families, and Appendix G includes a meaningful positional control for human-labeled verification: DeltaPRM exceeds its matched-coverage positional baseline by 9.2 AUROC points, with a reported interval excluding zero. These findings are stronger evidence of step-level utility than best-of-N alone.

Critical weaknesses

The principal efficiency contribution is not established for the implemented pipeline. The abstract and main tables advertise 278x and 217x labeling-cost reductions. Appendix I states that labeling as executed used full prefix forwards without reuse, costing 70-89x the fused KV-cache estimate on the agent pools. The cached formulation is plausible, and avoiding additional retrieval calls is a real advantage, but a cheapest-correct FLOPs projection is different from demonstrating the claimed economy in the system that produced the results. Fine-tuning also dominates the optimized DeltaPRM build cost (Table 23). Because efficiency is the main motivation relative to existing automatic labelers, I view this as a missing validation of a central contribution rather than a minor reporting issue. The manuscript needs to distinguish feasible optimized cost, actual labeling cost, and total construction cost; no misconduct inference follows from this discrepancy, which the appendix discloses. The method's distinctive supervision signal is not isolated from a strong confound. Appendix J specifies that the raw-log-likelihood arm uses MSE regression with no outcome auxiliary, whereas the delta/return arms use soft-label cross-entropy with that auxiliary. Removing the auxiliary from the return method lowers beam accuracy from 60.8% to 48.6% in Table 25, compared with 53.6% for raw likelihood. Thus the experiment does not establish that differencing, rather than the loss and outcome supervision, explains the advantage over a likelihood-level target. Outcome-only controls show that the full blended recipe is useful; they do not replace the missing matched likelihood-level control. Moreover, immediate gain and the default discounted return are not statistically separated in Table 25. A strictly evaluated method contribution therefore remains narrower than the paper's mechanism interpretation. The two different objectives named “lambda=0 ablation” further need to be disentangled.

Questions for authors

Clarification of the central efficiency claim: Which exact implementation generated the reported labels, and what are its HotpotQA labeling FLOPs and elapsed cost? Was the fused KV-cache implementation actually implemented and validated? Please distinguish as-run, standalone cached rescoring, and fused costs, and separate labeling cost from generation and fine-tuning cost. Assessment-relevant controlled comparison: Do you have a likelihood-level versus likelihood-gain comparison that holds the outcome auxiliary, loss family, scaling, and training settings fixed? This could change my assessment of the specific benefit of differencing. Please also map the two differently defined “lambda=0” variants to every reported ablation row. Clarification of evaluation splits and denominators: Is the 500-question HotpotQA set used to select gamma separate from the 500-question guided-decoding evaluation, which Table 7 calls validation? Please give the exact question counts and split identities for Table 1, especially MuSiQue and StrategyQA, since Appendix B/Table 10 describe original 500/140-question controls but some Table 1 strict-EM values do not match those denominators. Clarification of the main RL experiment: For Tables 3 and 18, is the PRM signal the discounted sum of score differences defined at the start of Appendix H, a raw sum of scores as stated in Table 18, or the mean score described later in Appendix H/Table 19? Is it applied through a separately normalized advantage or through a modified scalar reward? Please identify the exact configuration used for each table and explain how uncertainty accounts for training-seed variation, given the outcome-only HotpotQA scores of 45.2% and 58.1% across the two seeds.

10 comments
AuthorsOpenReview ↗

Response to Reviewer NaRv (1/6)

Dear Reviewer NaRv, thank you for your precise and constructive reading, including of the appendices. We provide the full computation table with measured costs, likelihood differencing ablation, additional RL training seed results, and address each of your provided weaknesses and questions.

Weakness 1 and Question 1: Efficiency of the implemented pipeline

We thank the reviewer for highlighting the distinction between the projected and executed labeling cost. Because efficiency is a central motivation of the paper, we agree that this claim is best supported by measured costs alongside the analytical estimate.

All labels used in the submission were generated by the as-run implementation, which runs a full forward pass over each prefix without prefix re-use. The 278× and 217× ratios are estimated and refer to the fused KV-cache formulation of the same label. They were computed analytically from the token census of our training pool, following the pricing convention of Appendix I (the computation is also detailed below Table R1).

At the time of submission, we had not yet implemented the fused KV-cache implementation. We therefore implemented it and measured costs for all three variants of our label computation (as-run, standalone with prefix reuse, and fused) and compared them against the MC and LLM-judge baselines and other outcome-supervised alternatives (Implicit PRM, PRIME, AutoPSV) requested in other reviews. The as-run, standalone, and fused implementations give the same likelihoods: every pair is correlated at 1.0, and any two differ by at most 0.34 nats per step on average.

For each of these methods, Table R1 reports the estimated labeling cost, the actual labeling cost, and the total PRM construction cost in exaFLOPs and measured GPU-hours.

Labeling cost. All label rules are run on one NVIDIA B200 over the same 1,000 HotpotQA questions × 8 rollouts (8,000 trajectories, 27,287 steps) and scaled to the paper’s 76,000-rollout pool (×9.5). The measured cost of the fused implementation (0.10 EF) closely matches the submission’s estimate (0.08 EF). Against it, MC estimation costs 233× the FLOPs and 22× the GPU-hours, and the per-step LLM judge (ThinkPRM) costs 181× and 19×. The as-run and standalone implementations cost 54× and 22× the fused FLOPs, or 4.0× and 2.4× its GPU-hours. The as-run implementation that produced the paper’s labels is also cheaper than both alternatives in measured cost. MC estimation costs 4.3× its FLOPs and 5.5× its GPU-hours, and ThinkPRM costs 3.4× and 4.7×. MC’s extra retrieval calls are not counted in either figure.

AuthorsOpenReview ↗

Response to Reviewer NaRv (2/6)

Total PRM construction cost. Training-data generation and PRM fine-tuning follow the standard PRM training procedure, and are identical for every method on a given task (2.96 EF / 1.25 GPU-h for data generation, 4.98 EF, 1.57 GPU-h for fine-tuning). The label computation is therefore the only stage that separates the methods. Including these shared costs, MC and LLM judge baselines are 4.0× and 3.3× the fused implementation in FLOPs, or 4.3× and 3.8× in GPU-h. We note that this fine-tuning cost is independent of the labeling method and falls outside the scope of this work. Reducing it is a separate question, addressed by work on efficient training.

Table R1 · Label cost per rule, 76,000 rollouts, one B200 in bf16 (EF = exaFLOPs)

Label ruleNfwdN_{\text{fwd}} per rolloutLabel EF, rule estimate (× fused)Label EF, actual (× fused)Label GPU-h, measured (× fused)Shared legs, measured, EF / GPU-hEnd-to-end, EF (× fused)End to end, GPU-h (× fused)
DeltaPRM (fused)(S + 1) · A0.08 (1×)0.10 (1×)0.53 (1×)7.94 / 2.828.0 (1×)3.35 (1×)
DeltaPRM (standalone)L′ + (S + 1) · A1.9 (24×)2.26 (22×)1.25 (2.4×)7.94 / 2.8210.2 (1.3×)4.1 (1.2×)
DeltaPRM (as-run)∑t(Lt+A)\sum_t (L_t + A)5.6 (70×)5.53 (54×)2.10 (4.0×)7.94 / 2.8213.5 (1.7×)4.9 (1.5×)
ORM (whole trajectory)0 (string match)0007.94 / 2.827.9 (1.0×)2.8 (0.8×)
Stepwise ORM0 (string match)0007.94 / 2.827.9 (1.0×)2.8 (0.8×)
Implicit PRML, inside its training loop1.8 (23×)—inside training7.96 / 5.979.8 (1.2×)6.0 (1.8×)
PRIME PRML per rollout per RL step1.6 (20×)—inside its RL run—not separable: no offline pool or fine-tune
AutoPSVL (one outcome-PRM read)1.8 (23×)2.62 (26×)1.27 (2.4×)12.9 / 8.2515.5 (1.9×)9.5 (2.8×)
MC Estimation (K=8)K · S · R / 2 (+ 6.3 retriever calls per step, unpriced)22.1 (278×)24.0 (233×)11.6 (22×)7.94 / 2.8231.9 (4.0×)14.4 (4.3×)
ThinkPRM (Qwen2.5-72B judge)L + (S + 1) · c, P = 72B18.5 (233×)18.6 (181×)9.85 (19×)7.94 / 2.8226.6 (3.3×)12.7 (3.8×)
VersaPRM (Llama-3.1-70B judge)L + c, P = 70B17.3 (217×)14.4 (140×)7.75 (15×)7.94 / 2.8222.3 (2.8×)10.6 (3.2×)
Agent-RRM (GPT-OSS-120B judge)L + c, c ≤ 1,024 decoded, P = 117B total28.6 (359×)31.1 (302×)5.36 (10×)7.94 / 2.8239.1 (4.9×)8.2 (2.4×)
RAG-Gym (GPT-4o)10 · R decoded + GPT-4o reads (unpriceable)≳16 + API (≳207×, floor)———not measurable on a GPU

How the columns are computed. Expected label EF applies the paper's rule (Appendix I: 2P FLOPs per forwarded token, cached prefixes free, judge verdicts floored at c = 20 tokens, retrieval for MC estimation in FLOPS is not accounted for) to the token census of the 1,000-question bench pool, with NfwdN_{\text{fwd}} the tokens each rule forwards per rollout (second column; S = 3.4 steps per rollout, L = 1,615 tokens per rollout, L′ the rollout tokens under the labeling prompt, R ≈ 1,420 response tokens, A = 15.7 tokens of answer elicitation plus gold answer, K Monte-Carlo continuations per step). Actual label EF counts the tokens each run forwarded, and GPU-h is wall-clock on one B200 for the 1,000 questions × 8 rollouts, both scaled by 9.5 to the full 76,000-rollout training pool. Shared legs are sampling the training prompts with the Search-R1 generator (2.96 EF, 1.25 GPU-h) and one epoch of the 7B PRM full fine-tune (4.98 EF, 1.57 GPU-h), paid by every rule; end-to-end adds both to the actual label cost.

We will replace the headline ratios in the abstract and main text with these measured figures. We will also state which implementation generated the reported labels, and report the estimated, actual, and end-to-end costs as separate quantities.

We appreciate this request for clarity. Separating these costs has made the efficiency claim considerably more precise and better supported.

AuthorsOpenReview ↗

Response to Reviewer NaRv (3/6)

Weakness 2 and Question 2: Isolating the supervision signal; the two “λ = 0” variants

Thank you for providing these clarifying points. Addressing them has made the manuscript more concrete. We reran the likelihood-level versus likelihood-gain comparison that isolates the effect of the target while holding the outcome auxiliary, loss, scaling, training recipe, and all other settings fixed, as shown in Table R2 below. In addition to the soft BCE loss used in the paper, we also compare the raw-log-likelihood target against the return target with the MSE loss. In the soft BCE loss, ℓt\ell_t is centered over the training pool so the label acts on a comparable scale.

Table R2 · Training target × loss, everything else fixed

Training targetLoss (outcome auxiliary has λ = 0.5)Best-of-N@256GreedyBeamMCTSAgentProcessBench AUROCGRPO (seeds)
Likelihood gain GtG_t (DeltaPRM)soft BCE0.4820.578†0.608†0.63278.459.8 (5)
Raw likelihood ℓt\ell_t (pool-centered)soft BCE0.4480.5880.6160.620†74.4†59.7 (2)
Stepwise ORMsoft BCE0.4080.4960.5080.52663.0–
ORM (whole trajectory)–0.476†0.5440.5700.59639.4–
Likelihood gain GtG_tMSE––0.5360.522––
Raw likelihood ℓt\ell_t (pool-centered)MSE––0.5360.554––

Bold: column best. †: not significantly worse than it (exact paired McNemar, α = 0.05; for AUROC, paired cluster bootstrap over the 50 questions). Strict exact match for best-of-N and guided decoding.

In the matched setting with soft BCE loss as in the paper, the raw likelihood performs on par with the likelihood gain on guided decoding (beam 0.616 vs 0.608, MCTS 0.620 vs 0.632) and as a GRPO reward (59.7 with two seeds vs 59.8 with five seeds), and slightly trails it on reranking (0.448 vs 0.482) and step verification (74.4 vs 78.4); Both remain ahead of the outcome-only PRMs trained on the same rollouts and backbone. With MSE, both targets fall to 0.536 under beam search, so the beam gap in Table 25 came mostly from the loss and the outcome auxiliary, rather than from differencing. We will replace the confounded row in Table 25 with this ablation that isolates the effect of differencing, correct the corresponding sentence in Sec. 5, and present ℓt\ell_t, δt\delta_t, and GtG_t in Sec. 3 as variants of one likelihood-based step target that perform comparably. We will also remove statements that attribute accuracy to differencing.

The core contribution is unaffected. The step target comes from the generator’s likelihood of the reference answer and needs only forward passes over existing trajectories, with no additional rollouts, retrieval calls, or judge model. Trained as a soft target together with outcome supervision, it yields a PRM that is competitive with published PRMs on guided search, step-level verification, and RL. All variants of the target share this.

We agree that the discount factor's effect is not evident in the multi-hop agentic setting. We included the discount to keep the labeling recipe flexible across domains, since how much a step's effect is delayed likely depends on the domain. In some domains, the discount mattered more than for retrieval agents; for example, the recipe uses γ = 0.8 for MATH, compared with γ = 0.3 for the multi-hop agentic domain. We additionally trained DeltaPRMs on MATH with the same recipe and γ = 0 or γ = 0.3. Averaged over two decoding seeds, MATH accuracy rises with γ under every decoder (Table R10). γ = 0.8 significantly outperforms both lower values under greedy and beam decoding, and under MCTS the differences are within noise.

Table R10 · Discount γ on MATH-500 (mean of 2 decoding seeds)

γGreedyBeam B=4MCTS-K16
0 (immediate gain)0.8160.8610.882†
0.30.8190.8700.885†
0.8 (paper)0.8460.8890.894

Bold: column best. †: not significantly worse than it (exact paired McNemar test on question × seed pairs, α = 0.05).

We conjecture the discount factor reflects how steps affect the answer likelihood in each domain. A retrieval step brings new external information into the context, so its effect tends to appear at that step. A step in a math solution adds no external information, and its effect may only appear once later steps build on it. We treat the propagation of future gains with γ as a tuned hyperparameter.

Reviewer_2XseOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DeltaPRM, which generates process supervision from the change in the correct answer’s log-likelihood between consecutive steps and trains a PRM for retrieval agents. The labeling strategy is simple and computationally attractive, and the guided decoding results demonstrate practical value.

Critical strengths

1 The paper proposes a simple and efficient process-labeling strategy that requires no additional rollouts, retrieval interactions, or external judge model. This is particularly valuable for retrieval agents, where sampling continuations involves costly environment interactions.

2 The trained PRM does not require the ground-truth answer at inference time and can support multiple uses, including best-of-N selection, guided decoding, and reinforcement learning.

3 The experiments cover several complementary evaluation settings. On HotpotQA, DeltaPRM improves MCTS accuracy from 59.6% with the ORM to 63.2%, and GRPO accuracy from 51.6% with outcome reward alone to 59.5%.

Critical weaknesses

1 Reliability of the supervision signal. An increase in the correct answer’s likelihood does not necessarily mean that the intermediate step is correct or useful. For example, misleading evidence containing the answer string could increase its likelihood, while a necessary intermediate retrieval may produce little immediate gain. Although Section 6 acknowledges this limitation, the human-labeled evaluation primarily assesses the trained PRM rather than the reliability of the original labels. Please directly evaluate the likelihood-gain labels against human judgments and analyze cases involving misleading passages, answer mentions without supporting evidence, and delayed contributions.

2 Inconsistent definitions of the ablations. According to Eq. (4), setting λ = 0 removes the outcome auxiliary and retains likelihood-based supervision. However, Appendix J also uses “λ = 0 ablation” to describe the opposite setting, where the likelihood target is replaced by the outcome label at every step. Table 25 distinguishes these settings and reports beam accuracies of 0.486 and 0.508, respectively, whereas Section 5 discusses the λ = 0 setting alongside Table 5, which reports 0.508. Please give these variants distinct names and clarify which results correspond to each objective. This directly affects the conclusion about the complementary contributions of process and outcome supervision.

3 The necessity of the discounted return is not clearly demonstrated. In Table 25, the immediate-gain variant with γ = 0 achieves 0.616 under beam search, compared with 0.608 for the proposed γ = 0.3 setting, without a significant difference. This supports the usefulness of the simpler signal but does not establish the benefit of propagating future gains. In addition, the undifferenced likelihood baseline changes both the target and the training objective, using regression without the outcome auxiliary. Please provide a more controlled comparison that holds the loss formulation and outcome supervision fixed, and clarify when the discounted target is preferable to the immediate gain.

4 The headline cost reduction needs clearer qualification. The 278× and 217× reductions use the fused KV-cache implementation. Appendix I states that standalone rescoring costs 24–37× more than the fused estimate, while the implementation actually run without prefix reuse costs 70–89× more. Although these distinctions are disclosed, they materially affect the practical interpretation of the headline numbers. Please clearly separate optimized estimates from measured implementation costs and report wall-clock labeling time under comparable hardware. An end-to-end comparison including trajectory generation and PRM training would further clarify the practical savings.

5 The evidence for improved process verification is mixed. Table 2 reports a favorable AUROC of 78.4, but the difference from Agent-RRM is not statistically significant under the reported test. Step accuracy is identical to Agent-RRM, and first-error accuracy is not significantly better than the baselines. These results support useful step ranking more clearly than reliable error localization. Please distinguish these capabilities and examine whether the proposed reward assigns high scores to incorrect intermediate steps that nevertheless increase answer likelihood.

Questions for authors

See Weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2Xse (1/5)

Dear Reviewer 2Xse, thank you for your detailed review. We address each weakness you provided with new experiments for improved precision of the manuscript.

Weakness 1 and Weakness 5: Reliability of the supervision signal and error localization behavior

Misleading passages, answer mentions without support, and delayed contributions. We consider the interesting failure case of misleading evidence that contains the answer and can increase its likelihood, and compare it with a relevant retrieval step. To study this behavior, we matched the 50 AgentProcessBench HotpotQA questions to their gold supporting facts and grouped the 361 non-final search steps (out of 644 steps in total) by what their retrieval returned and whether it contained the answer: a gold supporting paragraph that contains the answer (42%), a gold paragraph without the answer, typically the bridge hop (24%), a gold paragraph with the answer appearing only in another passage (4%), the answer string without any gold paragraph (4%), or neither (25%). Table R8 reports how well each score ranks each kind of retrieval above the uninformative searches that retrieve neither.

Table R8 · How each score ranks each kind of retrieval above uninformative searches (AgentProcessBench HotpotQA, non-final search steps; AUROC ×100 against the 91 searches that retrieve neither a gold paragraph nor the answer)

RetrievedStepsGtG_tTraining targetDeltaPRM
Gold paragraph containing the answer15362.278.581.3
Gold paragraph without the answer (bridge)8856.168.569.2
Gold paragraph, answer only in another passage1650.171.687.3
Answer string, no gold paragraph1365.783.086.9

The likelihood gain credits answer-bearing evidence weakly (GtG_t 62.2). Its failures follow the two cases you anticipated. It gives little credit to bridge retrievals, whose contribution is realized only at a later step (56.1). It credits passages that mention the answer without a gold paragraph more than any other group (65.7). The trained PRM partly recovers delayed contributions, separating bridge retrievals from uninformative steps at 69.2, and strongly credits answer-bearing evidence (81.3). Misleading mentions remain a limitation. We will report this failure analysis in more detail in the discussion.

Likelihood-gain labels on AgentProcessBench. In addition to evaluating human agreement for the trained PRM, we directly compare the likelihood-gain labels with human judgment in Table R6.

Table R6 · Labels, training target and trained scorers vs. human step labels (AgentProcessBench HotpotQA; 644 steps: the 734 benchmark steps without the 56 neutral steps and 34 steps beyond the PRM's context window; AUROC ×100)

ScoreAUROC
Discounted likelihood gain GtG_t49.1
Training target: σ(Gt/τ)\sigma(G_t/\tau) combined with outcome77.3
Trajectory outcome alone (oracle)80.1
Whole-trajectory ORM (trained)38.7
Stepwise ORM (trained; outcome label at every step)63.0
PRM trained on GtG_t (DeltaPRM)78.4

While the likelihood gain does not separate correct from incorrect steps on AgentProcessBench (GtG_t: 49.1 AUROC), the step-level signal combined with outcome auxiliary, as in our training target, reaches 77.3. The PRM trained on this target reaches 78.4, above its own training target, as training aggregates per-step signal noise over many trajectories. The oracle outcome reaches 80.1, reflecting the benchmark distribution, where incorrect steps are concentrated in failed trajectories. However, DeltaPRM’s combined signal of likelihood gain and the outcome auxiliary is not one that outcome supervision alone can learn. A stepwise ORM trained with the same recipe, but with only the outcome as its target at every step, reaches 63.0, and the whole-trajectory ORM, trained only on complete trajectories, ranks intermediate steps below chance (38.7).

Error localization behavior. We agree with the distinction between error localization and step ranking and adopt it. DeltaPRM is strong at ranking steps (AUROC 78.4) and at guiding search. At localizing the first error, it is only comparable to the baselines (first-error accuracy 61.6, step accuracy 68.4). Guided decoding, where DeltaPRM leads, relies on the ranking capability. We will state this distinction explicitly in Section 4.3. We would gently note that our central claim is not that DeltaPRM dominates existing verifiers on every metric. We claim that a PRM trained on labels obtained without additional rollouts, retrieval calls, or a judge model performs on par with verifiers built on far more expensive supervision.

AuthorsOpenReview ↗

Response to Reviewer 2Xse (2/5)

Incorrect steps that raise the answer likelihood. To evaluate this failure case, we used the 644 steps of the AgentProcessBench HotpotQA subset that are labelled correct or incorrect and fit in the PRM's context: 499 correct and 145 incorrect. For 88 of the 145 incorrect steps, the answer likelihood did not increase, as desired. For the other 57 of the incorrect steps, the likelihood increased, facing the failure case you described where the proposed reward assigns higher scores to incorrect intermediate steps that increase answer likelihood. In Table R11, for the two groups of incorrect steps whose likelihood rose and those that did not increase, we evaluate how often a correct step scores above an incorrect step from the group (AUROC) and how many incorrect steps in the group score above the median correct step, i.e., receive a score typical of a correct step.

Table R11 · Scores given to incorrect steps, by whether the answer likelihood rose at the step (AgentProcessBench HotpotQA; 644 steps: the 734 benchmark steps without the 56 neutral steps and 34 steps beyond the PRM's context window)

ScorerLikelihood rose (57 steps)Other incorrect steps (88 steps)
AUROC, correct vs. incorrect step (higher is better)
Discounted likelihood gain GtG_t21.866.8
Training target: σ(Gt/τ)\sigma(G_t/\tau) combined with outcome66.084.6
DeltaPRM73.181.9
Stepwise ORM62.463.3
Incorrect steps scored above the median correct step (lower is better)
Discounted likelihood gain GtG_t57 of 57 (100%)13 of 88 (15%)
Training target: σ(Gt/τ)\sigma(G_t/\tau) combined with outcome9 of 57 (16%)10 of 88 (11%)
DeltaPRM9 of 57 (16%)9 of 88 (10%)
Stepwise ORM22 of 57 (39%)31 of 88 (35%)

The raw likelihood gain GtG_t is misleading on the 57 steps where answer likelihood rose by construction: it scores all 57 above the median correct step (AUROC 21.8). Our training target, which combines GtG_t with the trajectory outcome, corrects most of these cases (66.0), and DeltaPRM, trained on this target, improves further (73.1) while scoring each step without seeing the outcome. The stepwise ORM, trained with the same recipe and data but with only the outcome as its target, reaches 62.4 AUROC on these incorrect steps where answer likelihood rose, and scores more than twice as many incorrect steps as good as typical correct ones (39% vs. 16%). Since the DeltaPRM and its stepwise ORM counterpart differ only in their target, this gap suggests the combined benefits that likelihood signal adds over outcome supervision.

We inspected the 57 incorrect steps where answer likelihood rose. 18% match the failure case where misleading evidence or a wrong entity leads the agent astray (9%), or a wrong final answer names the gold answer or reasoning that supports it (9%). 25% are unproductive searches, mostly with small gains. 19% decline to answer, either asking for clarification or stating that no information was found or that the question's premise is wrong. 14% are wrong answers with no visible link to the gold answer. 12% are answers that are arguably correct but marked incorrect (e.g., "Richmond River" for gold "Richmond"). 9% recall answer-relevant facts from memory before searching, mostly correctly. The remaining 3% are malformed search calls that the tool rejected with an error. Of the 9 incorrect steps that DeltaPRM scores like a typical correct step, four come from a single question with more than one valid answer (the gold's source calls "Kansas Song" a fight song of the University of Kansas, while the agents retrieve another University of Kansas fight song, "I'm a Jayhawk"), and two are misleading-evidence steps.

We will report this as a limitation in Section 6, alongside the label-level analysis of Weakness 1.

AuthorsOpenReview ↗

Response to Reviewer 2Xse (3/5)

Weakness 2 · Two objectives named “λ = 0 ablation”

Thank you for pointing out this discrepancy. In response, we revised the manuscript to name them distinctly, making the paper more concrete. λ = 0 keeps the likelihood-gain target and sets the weight of the outcome auxiliary to zero. Stepwise ORM replaces the target with the outcome label at every step, with no likelihood supervision.

In the original submission, the λ = 0 objective appears in three places: its description among the ablation arms of Appendix J, the Appendix J sentence that quotes its beam and best-of-256 accuracy, and the Table 25 row "no outcome aux (λ = 0)". Every other use of the name refers to the stepwise ORM: the second "λ = 0 ablation" row of Table 25, the "λ = 0 ablation" rows of Tables 5 and 24, the cost rows of Tables 21–23, and the remaining text of Appendices G and J. The Section 5 sentence named the λ = 0 objective but cited the stepwise ORM's result from Table 5; it now reads: removing the outcome term (λ = 0, 0.486) and training on the outcome label alone (stepwise ORM, 0.508). Removing either component lowers accuracy by 10 to 12 points, so the conclusion that the step-level likelihood signal and the outcome signal are complementary still holds.

Weakness 3 · Necessity of the discounted return; controlled raw-likelihood comparison

Effect of the discount factor. We agree that the discount factor's effect is not evident in the multi-hop agentic setting. We included the discount to keep the labeling recipe flexible across domains, since how much a step's effect is delayed likely depends on the domain. In some domains, the discount mattered more than for retrieval agents; for example, the recipe uses γ = 0.8 for MATH, compared with γ = 0.3 for the agentic domains. We additionally trained DeltaPRMs on MATH with the same recipe and γ = 0 or γ = 0.3. Averaged over two decoding seeds, MATH accuracy rises with γ under every decoder (Table R10). γ = 0.8 significantly outperforms both lower values under greedy and beam decoding, and under MCTS the differences are within noise.

Table R10 · Discount γ on MATH-500 (mean of 2 decoding seeds)

γGreedyBeam B=4MCTS-K16
0 (immediate gain)0.8160.8610.882†
0.30.8190.8700.885†
0.8 (paper)0.8460.8890.894

Bold: column best. †: not significantly worse than it (exact paired McNemar test on question × seed pairs, α = 0.05).

We conjecture that this reflects how steps affect the answer likelihood in each domain. A retrieval step brings new external information into the context, so its effect tends to appear at that step. A step in a math solution adds no external information, so its effect may appear only once later steps build on it.

Reviewer_iVm8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeltaPRM, which labels retrieval-agent steps using changes in the teacher-forced log-likelihood of the reference answer. The PRM learns discounted likelihood-gain targets together with outcome supervision, then scores trajectories without reference answers. Experiments cover reranking, guided decoding, human-labeled process verification, online RL, and transfer.

Critical strengths
  • The label construction is simple and requires no additional retrieval interactions. Distilling the likelihood-gain signal into a PRM makes it usable when reference answers are unavailable at inference.
  • Evaluation extends beyond best-of-N to guided decoding, human-labeled verification, cross-generator transfer, and online RL. On HotpotQA, MCTS accuracy improves from 59.6% with the matched ORM to 63.2% with DeltaPRM, suggesting the effectiveness of the learned score.
Critical weaknesses
  • Comparison with alternative inexpensive labels is incomplete. The matched ORM and ThinkPRM controls are useful, but the outcome-supervised PRM approaches discussed in Section 2, such as Implicit PRM and AutoPSV, are not directly compared. The raw-likelihood ablation also changes the loss to MSE and removes the outcome auxiliary. This limits attribution of the improvement specifically to likelihood differencing.
  • The headline efficiency ratios require qualification. The 278×/217× savings use a modeled fused-cache implementation. Appendix I states that the implementation actually run costs 70–89× this estimate on the agent pools. Estimated optimized labeling cost, actual labeling cost, and total construction cost should be separated before claiming the reported practical savings.
Questions for authors
  1. Can you compare with an outcome-supervised PRM baseline on the same rollouts/backbone, and compare raw likelihood versus likelihood gain with the same loss and outcome auxiliary?
  2. What labeling GPU-hours were measured, and what savings remain for the actual implementation and the complete PRM construction pipeline?
  3. Table 18 shows gains over outcome-only GRPO of 13.8 and 1.9 points across two seeds. Do additional seeds support the pooled 7.9-point estimate, with uncertainty over training runs?
5 comments
AuthorsOpenReview ↗

Response to Reviewer iVm8 (1/4)

Dear Reviewer iVm8, thank you for the careful reading. We ran the experiments you asked for, and the manuscript is more precise as a result. We provide results for additional inexpensive baselines, matched raw-likelihood ablation, full cost accounting, additional GRPO seeds, and address each of your provided weaknesses and questions.

Weakness 1 and Question 1 · Inexpensive outcome-derived baselines; the raw-likelihood ablation

Inexpensive baselines. We added the three outcome-derived baselines: implicit PRM (Yuan et al., 2024) (log-ratio of an outcome-trained policy to a reference), PRIME (Cui et al., 2025) (the same log-ratio learned online during RL), and AutoPSV (Lu et al., 2024) (step labels from drops in an outcome verifier's confidence). All three are trained on the same 76,000 Search-R1 rollouts and use the same 7B backbone as DeltaPRM. We also add an MC-estimation baseline: a PRM trained on the same rollouts and backbone, with step labels from K = 8 Monte-Carlo continuations per step (the Math-Shepherd rule). DeltaPRM significantly outperforms every inexpensive baseline while costing in labeling FLOPS 26× less (than AutoPSV) and matching or exceeding the MC baseline, whose labeling is 233× more expensive than DeltaPRM's (see the response to Weakness 2 and Question 2 below for full details on the estimated and measured computation). Among the newly added baselines, AutoPSV's guided decoding uses a single evaluation seed, and PRIME has a single GRPO seed. Implicit PRM and the MC baseline have the full three evaluation seeds and five GRPO seeds. We will add the remaining AutoPSV evaluation seeds in the final version.

Table R4 · Outcome-derived and MC-estimation baselines on HotpotQA, same rollouts and backbone

PRMGreedy / beam / MCTSBoN@256AgentProcessBench AUROCGRPO (seeds)
DeltaPRM.575† / .611† / .627.48278.4†59.8 (5)
Implicit PRM.497 / .503 / .473.32048.056.8 (5)
PRIME PRM.482 / .465 / .458.30653.243.4 (1)
AutoPSV.514 / .510 / .502.36865.3— (2, both runs collapsed)
MC K = 8 (Math-Shepherd rule).582 / .613 / .606.472†81.557.1 (5)

Bold: column best. †: not significantly worse than it (exact paired McNemar, α = 0.05; for AUROC, paired cluster bootstrap over the 50 questions; for GRPO, paired question-cluster bootstrap over seeds). Guided decoding is the mean of three evaluation seeds, except AutoPSV (one seed). GRPO is strict EM on 3,000 held-out HotpotQA questions.

AuthorsOpenReview ↗

Response to Reviewer iVm8 (2/4)

Likelihood differencing ablation. You are right that the submitted row changed target, loss, and auxiliary at once. We reran it changing only the target. Both PRMs now use the soft BCE target with the outcome auxiliary (λ = 0.5). ℓt\ell_t is centered over the training pool so that the label acts on a comparable scale. Table R2 reports this comparison together with MSE variants of both targets.

Table R2 · Training target × loss, everything else fixed

Training targetLoss (outcome auxiliary has λ = 0.5)Best-of-N@256GreedyBeamMCTSAgentProcessBench AUROCGRPO (seeds)
Likelihood gain GtG_t (DeltaPRM)soft BCE0.4820.578†0.608†0.63278.459.8 (5)
Raw likelihood ℓt\ell_t (pool-centered)soft BCE0.4480.5880.6160.620†74.4†59.7 (2)
Stepwise ORMsoft BCE0.4080.4960.5080.52663.0–
ORM (whole trajectory)–0.476†0.5440.5700.59639.4–
Likelihood gain GtG_tMSE––0.5360.522––
Raw likelihood ℓt\ell_t (pool-centered)MSE––0.5360.554––

Bold: column best. †: not significantly worse than it (exact paired McNemar, α = 0.05; for AUROC, paired cluster bootstrap over the 50 questions). Strict exact match for best-of-N and guided decoding.

Under the matched objective, the two targets perform on par. The raw likelihood-trained PRM matches the paper's PRM trained on the likelihood gain on guided decoding, matches it as a GRPO reward (59.7 with two seeds vs 59.8 with five seeds), and slightly trails it on reranking (0.448 vs 0.482) and step verification (74.4 vs 78.4); Both remain ahead of the outcome-only PRMs trained on the same rollouts and backbone. With MSE, both targets drop to 0.536 under beam search, so the beam gap in Table 25 is carried by the loss and the outcome auxiliary rather than by differencing.

We thank the reviewer for this point; the matched comparison clarifies which parts of the recipe drive the gains. We will replace the confounded row in Table 25 with the ablations of each loss with the outcome auxiliary, correct the corresponding sentence in Sec. 5, and present ℓt\ell_t, δt\delta_t, and GtG_t in Sec. 3 as variants of one likelihood-based step target that perform comparably. We will also remove statements that attribute accuracy to differencing.

The core contribution is unaffected. The step target comes from the generator’s likelihood of the reference answer and needs only forward passes over existing trajectories, with no additional rollouts, retrieval calls, or judge model. Trained as a soft target with outcome supervision, it yields a PRM competitive with published PRMs on guided search, step-level verification, and RL. All variants of the target share this.

Weakness 2 and Question 2 · Efficiency ratios need qualification; measured GPU-hours and remaining savings

We agree that the efficiency claim is best supported by measured costs. We therefore implement the optimized fused KV cache and provide the full compute table with estimated labeling cost, actual labeling cost, and total construction cost for 1) the as-run implementation in the paper, 2) the mentioned standalone implementation with prefix reuse, and 3) the optimized fused KV-cache implementation, alongside baselines and newly added outcome-supervised alternatives. The as-run, standalone, and fused implementations give the same likelihoods: every pair is correlated at 1.0, and any two differ by at most 0.34 nats per step on average.

Table R1 separates the estimated labeling cost, the actual labeling cost, and the total PRM construction cost, each in exaFLOPs and measured GPU-hours. We state below the table exactly how each column is computed.

Labeling cost. All label rules are run on one NVIDIA B200 over the same 1,000 HotpotQA questions × 8 rollouts (8,000 trajectories, 27,287 steps) and scaled to the paper’s 76,000-rollout pool (×9.5). Measured against our implemented fused labeler, MC estimation costs 233× the FLOPs and 22× the GPU-hours, and the per-step LLM judge (ThinkPRM) costs 181× the FLOPs and 19× the GPU-hours. The as-run and standalone implementations cost 54× and 22× more FLOPs than the fused implementation, or 4.0× and 2.4× more GPU-hours.

AuthorsOpenReview ↗

Response to Reviewer iVm8 (3/4)

Total PRM construction cost. Training-data generation and PRM fine-tuning cost are the same for every method on a given task, so label compute is what differentiates methods. Including these shared costs, MC and LLM judge baselines are 4.0× and 3.3× the fused implementation in FLOPs, or 4.3× and 3.8× in GPU-h.

Table R1 · Label cost per rule, 76,000 rollouts, one B200 in bf16 (EF = exaFLOPs)

Label ruleNfwdN_{\text{fwd}} per rolloutLabel EF, rule estimate (× fused)Label EF, actual (× fused)Label GPU-h, measured (× fused)Shared legs, measured, EF / GPU-hEnd-to-end, EF (× fused)End-to-end, GPU-h (× fused)
DeltaPRM (fused)(S + 1) · A0.08 (1×)0.10 (1×)0.53 (1×)7.94 / 2.828.0 (1×)3.35 (1×)
DeltaPRM (standalone)L′ + (S + 1) · A1.9 (24×)2.26 (22×)1.25 (2.4×)7.94 / 2.8210.2 (1.3×)4.1 (1.2×)
DeltaPRM (as-run)∑t(Lt+A)\sum_t (L_t + A)5.6 (70×)5.53 (54×)2.10 (4.0×)7.94 / 2.8213.5 (1.7×)4.9 (1.5×)
ORM (whole trajectory)0 (string match)0007.94 / 2.827.9 (1.0×)2.8 (0.8×)
Stepwise ORM0 (string match)0007.94 / 2.827.9 (1.0×)2.8 (0.8×)
Implicit PRML, inside its training loop1.8 (23×)—inside training7.96 / 5.979.8 (1.2×)6.0 (1.8×)
PRIME PRML per rollout per RL step1.6 (20×)—inside its RL run—not separable: no offline pool or fine-tune
AutoPSVL (one outcome-PRM read)1.8 (23×)2.62 (26×)1.27 (2.4×)12.9 / 8.2515.5 (1.9×)9.5 (2.8×)
MC Estimation (K=8)K · S · R / 2 (+ 6.3 retriever calls per step, unpriced)22.1 (278×)24.0 (233×)11.6 (22×)7.94 / 2.8231.9 (4.0×)14.4 (4.3×)
ThinkPRM (Qwen2.5-72B judge)L + (S + 1) · c, P = 72B18.5 (233×)18.6 (181×)9.85 (19×)7.94 / 2.8226.6 (3.3×)12.7 (3.8×)
VersaPRM (Llama-3.1-70B judge)L + c, P = 70B17.3 (217×)14.4 (140×)7.75 (15×)7.94 / 2.8222.3 (2.8×)10.6 (3.2×)
Agent-RRM (GPT-OSS-120B judge)L + c, c ≤ 1,024 decoded, P = 117B total28.6 (359×)31.1 (302×)5.36 (10×)7.94 / 2.8239.1 (4.9×)8.2 (2.4×)
RAG-Gym (GPT-4o)10 · R decoded + GPT-4o reads (unpriceable)≳16 + API (≳207×, floor)———not measurable on a GPU

How the columns are computed. Expected label EF applies the paper's rule (Appendix I: 2P FLOPs per forwarded token, cached prefixes free, judge verdicts floored at c = 20 tokens, retrieval for MC estimation in FLOPS is not accounted for) to the token census of the 1,000-question bench pool, with NfwdN_{\text{fwd}} the tokens each rule forwards per rollout (second column; S = 3.4 steps per rollout, L = 1,615 tokens per rollout, L′ the rollout tokens under the labeling prompt, R ≈ 1,420 response tokens, A = 15.7 tokens of answer elicitation plus gold answer, K Monte-Carlo continuations per step). Actual label EF counts the tokens each run forwarded, and GPU-h is wall-clock on one B200 for the 1,000 questions × 8 rollouts, both scaled by 9.5 to the full 76,000-rollout training pool. Shared legs are sampling the training prompts with the Search-R1 generator (2.96 EF, 1.25 GPU-h) and one epoch of the 7B PRM full fine-tune (4.98 EF, 1.57 GPU-h), paid by every rule; end-to-end adds both to the actual label cost.

In the revision, we will replace the headline ratios with these measured figures. We will also report the estimated, actual, and end-to-end costs separately in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.