- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DeltaPRM, a process reward model for retrieval agents trained using changes in the frozen generator's teacher-forced log-likelihood of a reference answer between successive reasoning/search steps. Discounted sums of these changes are converted to soft labels, and the PRM is trained jointly against these labels and trajectory-level outcome labels. Label construction requires no additional sampled continuations, retrieval calls, or external judge, although the training trajectories themselves must first be generated. After training, the PRM can score prefixes without reference answers. Experiments cover best-of-N selection, guided decoding, human-labeled step verification, and GRPO, with additional generator and domain transfer. The strongest evidence is on multi-hop search; results outside this setting are often comparable to established PRMs rather than uniformly superior.
Critical strengths
A practical supervision mechanism that avoids additional environment interaction. Reference-answer likelihood differences provide a graded, signed signal from supervision already present in QA data. Distilling that signal into a prefix scorer removes the need for reference answers at deployment. This is a consequential construction for retrieval agents, where continuation-based labeling entails expensive search interaction. Substantial evidence of useful guidance on multi-hop search. On HotpotQA, DeltaPRM reaches 60.8% under beam search and 63.2% under MCTS, versus 57.0% and 59.6% for the matched-backbone ORM (Table 1). The prefix-broadcast outcome control is also weaker (Tables 5 and 25). The paper evaluates multiple datasets and generator families, and Appendix G includes a meaningful positional control for human-labeled verification: DeltaPRM exceeds its matched-coverage positional baseline by 9.2 AUROC points, with a reported interval excluding zero. These findings are stronger evidence of step-level utility than best-of-N alone.
Critical weaknesses
The principal efficiency contribution is not established for the implemented pipeline. The abstract and main tables advertise 278x and 217x labeling-cost reductions. Appendix I states that labeling as executed used full prefix forwards without reuse, costing 70-89x the fused KV-cache estimate on the agent pools. The cached formulation is plausible, and avoiding additional retrieval calls is a real advantage, but a cheapest-correct FLOPs projection is different from demonstrating the claimed economy in the system that produced the results. Fine-tuning also dominates the optimized DeltaPRM build cost (Table 23). Because efficiency is the main motivation relative to existing automatic labelers, I view this as a missing validation of a central contribution rather than a minor reporting issue. The manuscript needs to distinguish feasible optimized cost, actual labeling cost, and total construction cost; no misconduct inference follows from this discrepancy, which the appendix discloses. The method's distinctive supervision signal is not isolated from a strong confound. Appendix J specifies that the raw-log-likelihood arm uses MSE regression with no outcome auxiliary, whereas the delta/return arms use soft-label cross-entropy with that auxiliary. Removing the auxiliary from the return method lowers beam accuracy from 60.8% to 48.6% in Table 25, compared with 53.6% for raw likelihood. Thus the experiment does not establish that differencing, rather than the loss and outcome supervision, explains the advantage over a likelihood-level target. Outcome-only controls show that the full blended recipe is useful; they do not replace the missing matched likelihood-level control. Moreover, immediate gain and the default discounted return are not statistically separated in Table 25. A strictly evaluated method contribution therefore remains narrower than the paper's mechanism interpretation. The two different objectives named “lambda=0 ablation” further need to be disentangled.
Questions for authors
Clarification of the central efficiency claim: Which exact implementation generated the reported labels, and what are its HotpotQA labeling FLOPs and elapsed cost? Was the fused KV-cache implementation actually implemented and validated? Please distinguish as-run, standalone cached rescoring, and fused costs, and separate labeling cost from generation and fine-tuning cost. Assessment-relevant controlled comparison: Do you have a likelihood-level versus likelihood-gain comparison that holds the outcome auxiliary, loss family, scaling, and training settings fixed? This could change my assessment of the specific benefit of differencing. Please also map the two differently defined “lambda=0” variants to every reported ablation row. Clarification of evaluation splits and denominators: Is the 500-question HotpotQA set used to select gamma separate from the 500-question guided-decoding evaluation, which Table 7 calls validation? Please give the exact question counts and split identities for Table 1, especially MuSiQue and StrategyQA, since Appendix B/Table 10 describe original 500/140-question controls but some Table 1 strict-EM values do not match those denominators. Clarification of the main RL experiment: For Tables 3 and 18, is the PRM signal the discounted sum of score differences defined at the start of Appendix H, a raw sum of scores as stated in Table 18, or the mean score described later in Appendix H/Table 19? Is it applied through a separately normalized advantage or through a modified scalar reward? Please identify the exact configuration used for each table and explain how uncertainty accounts for training-seed variation, given the outcome-only HotpotQA scores of 45.2% and 58.1% across the two seeds.
10 comments
Response to Reviewer NaRv (2/6)
Total PRM construction cost. Training-data generation and PRM fine-tuning follow the standard PRM training procedure, and are identical for every method on a given task (2.96 EF / 1.25 GPU-h for data generation, 4.98 EF, 1.57 GPU-h for fine-tuning). The label computation is therefore the only stage that separates the methods. Including these shared costs, MC and LLM judge baselines are 4.0× and 3.3× the fused implementation in FLOPs, or 4.3× and 3.8× in GPU-h. We note that this fine-tuning cost is independent of the labeling method and falls outside the scope of this work. Reducing it is a separate question, addressed by work on efficient training.
Table R1 · Label cost per rule, 76,000 rollouts, one B200 in bf16 (EF = exaFLOPs)
| Label rule | per rollout | Label EF, rule estimate (× fused) | Label EF, actual (× fused) | Label GPU-h, measured (× fused) | Shared legs, measured, EF / GPU-h | End-to-end, EF (× fused) | End to end, GPU-h (× fused) |
|---|---|---|---|---|---|---|---|
| DeltaPRM (fused) | (S + 1) · A | 0.08 (1×) | 0.10 (1×) | 0.53 (1×) | 7.94 / 2.82 | 8.0 (1×) | 3.35 (1×) |
| DeltaPRM (standalone) | L′ + (S + 1) · A | 1.9 (24×) | 2.26 (22×) | 1.25 (2.4×) | 7.94 / 2.82 | 10.2 (1.3×) | 4.1 (1.2×) |
| DeltaPRM (as-run) | 5.6 (70×) | 5.53 (54×) | 2.10 (4.0×) | 7.94 / 2.82 | 13.5 (1.7×) | 4.9 (1.5×) | |
| ORM (whole trajectory) | 0 (string match) | 0 | 0 | 0 | 7.94 / 2.82 | 7.9 (1.0×) | 2.8 (0.8×) |
| Stepwise ORM | 0 (string match) | 0 | 0 | 0 | 7.94 / 2.82 | 7.9 (1.0×) | 2.8 (0.8×) |
| Implicit PRM | L, inside its training loop | 1.8 (23×) | — | inside training | 7.96 / 5.97 | 9.8 (1.2×) | 6.0 (1.8×) |
| PRIME PRM | L per rollout per RL step | 1.6 (20×) | — | inside its RL run | — | not separable: no offline pool or fine-tune | |
| AutoPSV | L (one outcome-PRM read) | 1.8 (23×) | 2.62 (26×) | 1.27 (2.4×) | 12.9 / 8.25 | 15.5 (1.9×) | 9.5 (2.8×) |
| MC Estimation (K=8) | K · S · R / 2 (+ 6.3 retriever calls per step, unpriced) | 22.1 (278×) | 24.0 (233×) | 11.6 (22×) | 7.94 / 2.82 | 31.9 (4.0×) | 14.4 (4.3×) |
| ThinkPRM (Qwen2.5-72B judge) | L + (S + 1) · c, P = 72B | 18.5 (233×) | 18.6 (181×) | 9.85 (19×) | 7.94 / 2.82 | 26.6 (3.3×) | 12.7 (3.8×) |
| VersaPRM (Llama-3.1-70B judge) | L + c, P = 70B | 17.3 (217×) | 14.4 (140×) | 7.75 (15×) | 7.94 / 2.82 | 22.3 (2.8×) | 10.6 (3.2×) |
| Agent-RRM (GPT-OSS-120B judge) | L + c, c ≤ 1,024 decoded, P = 117B total | 28.6 (359×) | 31.1 (302×) | 5.36 (10×) | 7.94 / 2.82 | 39.1 (4.9×) | 8.2 (2.4×) |
| RAG-Gym (GPT-4o) | 10 · R decoded + GPT-4o reads (unpriceable) | ≳16 + API (≳207×, floor) | — | — | — | not measurable on a GPU |
How the columns are computed. Expected label EF applies the paper's rule (Appendix I: 2P FLOPs per forwarded token, cached prefixes free, judge verdicts floored at c = 20 tokens, retrieval for MC estimation in FLOPS is not accounted for) to the token census of the 1,000-question bench pool, with the tokens each rule forwards per rollout (second column; S = 3.4 steps per rollout, L = 1,615 tokens per rollout, L′ the rollout tokens under the labeling prompt, R ≈ 1,420 response tokens, A = 15.7 tokens of answer elicitation plus gold answer, K Monte-Carlo continuations per step). Actual label EF counts the tokens each run forwarded, and GPU-h is wall-clock on one B200 for the 1,000 questions × 8 rollouts, both scaled by 9.5 to the full 76,000-rollout training pool. Shared legs are sampling the training prompts with the Search-R1 generator (2.96 EF, 1.25 GPU-h) and one epoch of the 7B PRM full fine-tune (4.98 EF, 1.57 GPU-h), paid by every rule; end-to-end adds both to the actual label cost.
We will replace the headline ratios in the abstract and main text with these measured figures. We will also state which implementation generated the reported labels, and report the estimated, actual, and end-to-end costs as separate quantities.
We appreciate this request for clarity. Separating these costs has made the efficiency claim considerably more precise and better supported.
Response to Reviewer NaRv (3/6)
Weakness 2 and Question 2: Isolating the supervision signal; the two “λ = 0” variants
Thank you for providing these clarifying points. Addressing them has made the manuscript more concrete. We reran the likelihood-level versus likelihood-gain comparison that isolates the effect of the target while holding the outcome auxiliary, loss, scaling, training recipe, and all other settings fixed, as shown in Table R2 below. In addition to the soft BCE loss used in the paper, we also compare the raw-log-likelihood target against the return target with the MSE loss. In the soft BCE loss, is centered over the training pool so the label acts on a comparable scale.
Table R2 · Training target × loss, everything else fixed
| Training target | Loss (outcome auxiliary has λ = 0.5) | Best-of-N@256 | Greedy | Beam | MCTS | AgentProcessBench AUROC | GRPO (seeds) |
|---|---|---|---|---|---|---|---|
| Likelihood gain (DeltaPRM) | soft BCE | 0.482 | 0.578† | 0.608† | 0.632 | 78.4 | 59.8 (5) |
| Raw likelihood (pool-centered) | soft BCE | 0.448 | 0.588 | 0.616 | 0.620† | 74.4† | 59.7 (2) |
| Stepwise ORM | soft BCE | 0.408 | 0.496 | 0.508 | 0.526 | 63.0 | – |
| ORM (whole trajectory) | – | 0.476† | 0.544 | 0.570 | 0.596 | 39.4 | – |
| Likelihood gain | MSE | – | – | 0.536 | 0.522 | – | – |
| Raw likelihood (pool-centered) | MSE | – | – | 0.536 | 0.554 | – | – |
Bold: column best. †: not significantly worse than it (exact paired McNemar, α = 0.05; for AUROC, paired cluster bootstrap over the 50 questions). Strict exact match for best-of-N and guided decoding.
In the matched setting with soft BCE loss as in the paper, the raw likelihood performs on par with the likelihood gain on guided decoding (beam 0.616 vs 0.608, MCTS 0.620 vs 0.632) and as a GRPO reward (59.7 with two seeds vs 59.8 with five seeds), and slightly trails it on reranking (0.448 vs 0.482) and step verification (74.4 vs 78.4); Both remain ahead of the outcome-only PRMs trained on the same rollouts and backbone. With MSE, both targets fall to 0.536 under beam search, so the beam gap in Table 25 came mostly from the loss and the outcome auxiliary, rather than from differencing. We will replace the confounded row in Table 25 with this ablation that isolates the effect of differencing, correct the corresponding sentence in Sec. 5, and present , , and in Sec. 3 as variants of one likelihood-based step target that perform comparably. We will also remove statements that attribute accuracy to differencing.
The core contribution is unaffected. The step target comes from the generator’s likelihood of the reference answer and needs only forward passes over existing trajectories, with no additional rollouts, retrieval calls, or judge model. Trained as a soft target together with outcome supervision, it yields a PRM that is competitive with published PRMs on guided search, step-level verification, and RL. All variants of the target share this.
We agree that the discount factor's effect is not evident in the multi-hop agentic setting. We included the discount to keep the labeling recipe flexible across domains, since how much a step's effect is delayed likely depends on the domain. In some domains, the discount mattered more than for retrieval agents; for example, the recipe uses γ = 0.8 for MATH, compared with γ = 0.3 for the multi-hop agentic domain. We additionally trained DeltaPRMs on MATH with the same recipe and γ = 0 or γ = 0.3. Averaged over two decoding seeds, MATH accuracy rises with γ under every decoder (Table R10). γ = 0.8 significantly outperforms both lower values under greedy and beam decoding, and under MCTS the differences are within noise.
Table R10 · Discount γ on MATH-500 (mean of 2 decoding seeds)
| γ | Greedy | Beam B=4 | MCTS-K16 |
|---|---|---|---|
| 0 (immediate gain) | 0.816 | 0.861 | 0.882† |
| 0.3 | 0.819 | 0.870 | 0.885† |
| 0.8 (paper) | 0.846 | 0.889 | 0.894 |
Bold: column best. †: not significantly worse than it (exact paired McNemar test on question × seed pairs, α = 0.05).
We conjecture the discount factor reflects how steps affect the answer likelihood in each domain. A retrieval step brings new external information into the context, so its effect tends to appear at that step. A step in a math solution adds no external information, and its effect may only appear once later steps build on it. We treat the propagation of future gains with γ as a tuned hyperparameter.
Response to Reviewer NaRv (1/6)
Dear Reviewer NaRv, thank you for your precise and constructive reading, including of the appendices. We provide the full computation table with measured costs, likelihood differencing ablation, additional RL training seed results, and address each of your provided weaknesses and questions.
Weakness 1 and Question 1: Efficiency of the implemented pipeline
We thank the reviewer for highlighting the distinction between the projected and executed labeling cost. Because efficiency is a central motivation of the paper, we agree that this claim is best supported by measured costs alongside the analytical estimate.
All labels used in the submission were generated by the as-run implementation, which runs a full forward pass over each prefix without prefix re-use. The 278× and 217× ratios are estimated and refer to the fused KV-cache formulation of the same label. They were computed analytically from the token census of our training pool, following the pricing convention of Appendix I (the computation is also detailed below Table R1).
At the time of submission, we had not yet implemented the fused KV-cache implementation. We therefore implemented it and measured costs for all three variants of our label computation (as-run, standalone with prefix reuse, and fused) and compared them against the MC and LLM-judge baselines and other outcome-supervised alternatives (Implicit PRM, PRIME, AutoPSV) requested in other reviews. The as-run, standalone, and fused implementations give the same likelihoods: every pair is correlated at 1.0, and any two differ by at most 0.34 nats per step on average.
For each of these methods, Table R1 reports the estimated labeling cost, the actual labeling cost, and the total PRM construction cost in exaFLOPs and measured GPU-hours.
Labeling cost. All label rules are run on one NVIDIA B200 over the same 1,000 HotpotQA questions × 8 rollouts (8,000 trajectories, 27,287 steps) and scaled to the paper’s 76,000-rollout pool (×9.5). The measured cost of the fused implementation (0.10 EF) closely matches the submission’s estimate (0.08 EF). Against it, MC estimation costs 233× the FLOPs and 22× the GPU-hours, and the per-step LLM judge (ThinkPRM) costs 181× and 19×. The as-run and standalone implementations cost 54× and 22× the fused FLOPs, or 4.0× and 2.4× its GPU-hours. The as-run implementation that produced the paper’s labels is also cheaper than both alternatives in measured cost. MC estimation costs 4.3× its FLOPs and 5.5× its GPU-hours, and ThinkPRM costs 3.4× and 4.7×. MC’s extra retrieval calls are not counted in either figure.