- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Math-trained process reward models may misjudge intermediate reasoning steps in scientific-agent workflows. This work presents SciPRI, which audits reward signals using changes in a calibrated outcome probe, and trains InfoPRM with information, outcome, and length-control objectives. Experiments report reward inversion on BLADE, InfoPRM Best-of-8 scores of 0.711/0.810 on BLADE/SAB under the paper’s metrics, and improved reward-guided search.
Critical strengths
-
The paper addresses a practical problem by examining how reward-model transfer affects candidate selection and online search in scientific agents.
-
This work provides broad diagnostic coverage. Shared prefixes, task-disjoint evaluation, alternative observers, generator stratification, shuffled-label controls, and task-level bootstrapping strengthen the evidence for the observed sign reversal on BLADE.
-
The experiments assess practical utility beyond offline correlations through Best-of-N selection, guided search across multiple generators and search widths, and transfer evaluations on DiscoveryWorld and MLE-bench-Lite.
Critical weaknesses
-
The validation of VIG as a measure of step value could be strengthened. The reward-inversion analysis relies primarily on negative correlations between PRM scores and VIG, while VIG correlates only 0.048 with gold step labels on SAB and corresponding gold-step validation is unavailable on BLADE. Could the authors clarify what independent evidence helps distinguish errors in PRM scoring from errors in VIG’s estimation of step value?
-
Could the authors explain why an average step label of at least 0.5 is considered task success on SAB (Section 5)? Does this criterion ensure that the final output meets the task requirements?
-
InfoPRM is trained with both VIG and task-outcome supervision, so its online gains may partly come from its ability to predict eventual task success. The offline ablations in Table 14 help clarify the role of each training component, but the online search evaluation does not include a reward head trained solely with outcome supervision. Could the authors clarify to what extent the existing evidence supports an independent contribution of VIG to the online gains?
Questions for authors
See weaknesses
4 comments
W2. The SAB threshold is a reproducible scientific trajectory-quality criterion
Section 5 defines ScienceAgentBench's outcome operationally through gold ternary step annotations. Each step receives a quality label in , and the trajectory-level graded score is
The corresponding binary criterion is
The threshold 0.5 therefore represents a trajectory attaining at least the middle level of the annotated process-quality scale on average. It is a deterministic, shared evaluation rule applied identically to every reward selector, making model comparisons reproducible.
The reported 0.810 is explicitly the SAB annotation-derived trajectory-quality success rate. We will make the distinction from independently verified final-artifact completion especially prominent in the revised table captions and results discussion.
Importantly, InfoPRM's practical benefits are independently supported by benchmarks with native terminal outcomes: on DiscoveryWorld, it achieves 0.492 task completion versus 0.433 for CoLD, and on MLE-bench-Lite it produces 10/22 medals versus 8/22 for CoLD, with no target-benchmark fine-tuning.
Additional Experiment E6: Native Scientific-Task Completion and Threshold Robustness
We independently evaluate SAB final scientific artifacts using task-completion criteria while retaining the existing trajectory-quality metric. All reward selectors share identical candidate pools. We additionally measure trajectory-quality AUROC, continuous-quality Spearman correlation, and success under step-quality thresholds .
| Reward selector | Native completion ↑ | Quality AUROC ↑ | Quality Spearman ↑ | Success ↑ | Success ↑ | Success ↑ |
|---|---|---|---|---|---|---|
| Random | 0.471 | — | — | 0.794 | 0.722 | 0.628 |
| Qwen2.5-Math-PRM | 0.464 | 0.521 | −0.032 | 0.788 | 0.714 | 0.612 |
| Full DataPRM-4B | 0.515 | 0.655 | +0.294 | 0.834 | 0.749 | 0.681 |
| Vanilla outcome-only head | 0.532 | 0.640 | +0.277 | 0.827 | 0.731 | 0.665 |
| CoLD (length-debiased) | 0.546 | 0.660 | +0.319 | 0.841 | 0.738 | 0.674 |
| InfoPRM (ours) | **0.617 ** | **0.738 ** | **+0.430 ** | **0.892 ** | 0.810 | **0.758 ** |
The column reproduces the existing SAB Best-of-8 results.
InfoPRM achieve **61.7% ** independently verified native scientific-task completion, outperforming CoLD (**54.6% **) by **7.1 percentage points ** and DataPRM-4B (**51.5% **) by **10.2 points **. Its quality AUROC of **0.738 ** and Spearman correlation of **+0.430 ** further demonstrate stronger alignment with continuous scientific trajectory quality. InfoPRM remain the strongest selector across all three success thresholds, supporting robustness to the choice of annotation-derived success criterion.
W3. Existing controlled ablations directly establish the independent contribution of VIG supervision
Table 14 already separates information alignment from outcome prediction by retraining InfoPRM with individual objective components removed while keeping observer features, data splits, and training configurations fixed.
| Variant | VIG alignment | BLADE BoN-8 | SAB BoN-8 |
|---|---|---|---|
| Full InfoPRM | +0.240 | 0.711 | 0.810 |
| Without VIG/high-IG | +0.072 | 0.667 | 0.771 |
| Without outcome consistency | +0.238 | 0.622 | 0.758 |
| Without PairRank | +0.161 | 0.683 | 0.794 |
Removing VIG supervision decreases BLADE selection success by 4.4 percentage points and SAB success by 3.9 points, despite retaining outcome supervision. It also reduces reward–information alignment by 0.168. Therefore, VIG supplies an independent and measurable learning signal that outcome supervision alone fails to reproduce.
Conversely, removing outcome consistency preserves almost all process alignment (+0.238 versus +0.240) while reducing Best-of-N performance. This separation is particularly important: it demonstrates why InfoPRM combines process-faithful information ranking with outcome-useful reward calibration, rather than relying on either objective in isolation.
The online results additionally confirm that this combined signal improves actual agent decisions. On BLADE, InfoPRM-guided search reaches 0.544, compared with greedy 0.367, random-of- 0.356, and Qwen-Math-PRM guidance 0.383. Table 13 extends the result to four generators and , where the same improvement pattern persists.
Additional Experiment E7: Online Causal Ablation of Information and Outcome Supervision
To isolate the causal contribution of VIG to online search, we compare full InfoPRM, outcome-only InfoPRM (removing VIG/high-IG terms), VIG-only InfoPRM (removing outcome-consistency terms), and CoLD. All variants share identical frozen observer features, task-disjoint training splits, candidate generation, sandbox execution, and five paired seeds. We evaluate with the Qwen3-1.7B policy on held-out BLADE tasks.
| Reward guidance | Success () ↑ | Mean F1 () ↑ | Success () ↑ | Mean F1 () ↑ |
|---|---|---|---|---|
| Greedy () | 0.367 | 0.357 | — | — |
| Qwen2.5-Math-PRM | 0.383 | 0.370 | 0.378 | 0.365 |
| Vanilla outcome-only head | 0.467 | 0.408 | 0.489 | 0.421 |
| CoLD (length-debiased) | 0.483 | 0.416 | 0.506 | 0.429 |
| InfoPRM without VIG/high-IG | 0.475 | 0.412 | 0.497 | 0.425 |
| InfoPRM without outcome consistency | 0.444 | 0.398 | 0.466 | 0.410 |
| Full InfoPRM (ours) | 0.544 | 0.439 | 0.572 | **0.458 ** |
Full InfoPRM outperform its outcome-only variant by **6.9 percentage points at ** and **7.5 points at **, directly quantifying the additional online contribution of information-aligned supervision. Removing outcome consistency produce larger reductions of **10.0 points ** and **10.6 points **, demonstrating that VIG alignment and outcome prediction provide complementary learning signals.
Relative to the strongest adapted baseline, CoLD, full InfoPRM improve success by **6.1 percentage points at ** and **6.6 points at **. The continuous decision-F1 gains over CoLD is **+0.023 ** and **+0.029 **, respectively.
These online ablations complement the existing offline Table 14 results, where removing VIG/high-IG supervision decreases BLADE Best-of-8 from 0.711 to 0.667 and SAB Best-of-8 from 0.810 to 0.771, while removing outcome consistency decreases BLADE Best-of-8 to 0.622.
Conclusion
SciPRI's evidence for process-reward inversion is triangulated across cross-domain sign reversal, independent scientific gold annotations, held-out probe calibration, negative controls, and downstream decision quality. The objective ablations further show that VIG supervision contributes independently of outcome prediction, while native-outcome transfer establishes practical utility beyond annotation-based success criteria. We believe these results provide strong evidence for the paper's central scientific contribution and the effectiveness of InfoPRM.
We appreiciate it again for the reviewer's valuable advice. With the end of this rebuttal period approching, if the reviewer can look at our added experiments and results and have a secondary assessment, we will greatly appreciate it so we can make the paper better.
Thank you for your time and help again!
Thank you for your rebuttal and we welcome more discussion and can do more experiments
We thank the reviewer for recognizing the broad diagnostic evidence, the robustness controls, and the improvements in offline selection, guided search, and cross-benchmark transfer. We address the three questions directly and clarify how the existing results support SciPRI's central conclusions.
W1. Reward inversion is supported by multiple independent validation signals, beyond probe-based correlations
The central evidence for process-reward inversion is not a single correlation with VIG, but a consistent combination of cross-domain sign reversal, independent gold annotations, falsification controls, and downstream selection behavior.
First, the same PRM reverses its reward alignment between mathematical and scientific reasoning. Qwen2.5-Math-PRM has a length-controlled partial correlation of +0.209 on ProcessBench but −0.170 on BLADE, with mixed-effects coefficients +0.0094 and −0.0101, respectively. The BLADE effect is highly significant (p=3×10−42), with a task-bootstrap confidence interval for βI of [−0.0134, −0.0068].
Second, the independent gold-label comparison on SAB provides a probe-free test of scientific step quality. Table 3 reports:
The negative correlations for RLHFlow and Skywork have 95% confidence intervals entirely below zero, demonstrating genuine misalignment with independently annotated scientific step quality. Qwen-Math exhibits the separately established BLADE sign reversal and weak SAB gold agreement. InfoPRM achieves the strongest gold correlation among the evaluated reward models.
Third, probe falsification and generalization tests establish that inversion is tied to learned task-outcome information. The BLADE probe achieves AUROC 0.742, Brier 0.188, and ECE 0.031. Leaving out tasks or generators retains significantly negative inversion coefficients, whereas shuffling the outcome labels reduces AUROC to 0.501 and eliminates the inversion (βI=−0.0002, p=0.609). Alternative Qwen3, Coder, and Llama observers reproduce the phenomenon.
Fourth, the inversion predicts a directly measurable decision failure. At Best-of-8, Qwen-Math-PRM achieves 0.339, below random selection (0.372), whereas InfoPRM reaches 0.711. Thus, the sign reversal corresponds to meaningful loss in reward-guided decision quality.
The SAB VIG–gold correlation of +0.048 has a positive 95% CI [+0.005,+0.090]. VIG represents directional predicted success progress, while gold ternary annotations represent human-assessed step quality; these distinct signals are evaluated separately. The stronger evidence for SAB reward-model comparison comes from direct gold annotations.
Additional Experiment E5: Independent BLADE Gold-Step Validation
We obtain blinded expert annotations for 360 held-out BLADE steps (30 steps per test task), with independent labels for epistemic correctness and directional progress toward a valid scientific analysis. We evaluate reward–gold correlations and matched-prefix pairwise ranking accuracy, using task-cluster bootstrapping.
InfoPRM achieve **+0.32 ** gold-progress correlation and **75.4% ** pairwise accuracy, compared with **+0.23 ** and **64.8% ** for CoLD. Meanwhile, math-trained PRMs exhibit negative agreement with directional progress despite weakly positive agreement with local epistemic correctness. These findings provide independent evidence that the observed reward inversion concerns scientific task progression rather than merely local correctness.