acceptodds
Under review as a conference paper at ICLR 2027

SciPRI: Process-Reward Inversion in Scientific Agents

Abstract

Process reward models (PRMs) score the intermediate steps of language-model agents and increasingly drive step selection, tree search, and reinforcement learning. Almost all public PRMs are trained on mathematical reasoning, where a step’s value is its local arithmetic/logical correctness. Scientific agents (data analysis, experiment design, ML engineering) instead produce steps whose value is the information they contribute toward a correct scientific conclusion, often carried by terse, information-dense actions (code, observations, exploratory tests) rather than fluent prose. We introduce SciPRI, a measurement-and-mitigation framework that quantifies per-step information gain 〖VIG〗_t with a calibrated probe over frozen hidden states (the change in the calibrated log-probability of task success) and uses it to audit reward models. Our central finding is process-reward inversion: the most widely used PRMs, math-trained discriminative reward models, become negatively aligned with information gain on scientific steps. The same model flips sign between domains: Qwen2.5-Math-PRM has length-controlled partial correlation ρ=+0.209 and mixed-effects slope β_I=+0.0094 (p< 10^(-29)) between reward and information gain on a math control, but ρ=−0.170, β_I=−0.0101 (p=3 × 10^(-42)) on scientific tasks. The inversion is gold-validated on 102 ScienceAgentBench tasks (math PRMs correlate −0.132 to +0.045 with gold scientific step quality, where they should be strongly positive), is robust across six generator models, three alternative observer models, and many estimator choices, and harms Best-of-N selection below random. A panel of reward models reveals a spectrum of failure: discriminative math PRMs invert, value-head PRMs collapse, and a strong general LLM-critic mis-selects. We introduce InfoPRM, a lightweight reward head trained to track information gain and predict the outcome while remaining length-decorrelated; it matches the best Best-of-N result on BLADE (0.711) and wins outright on ScienceAgentBench (0.810 vs. 0.749), agrees with gold scientific step labels (+0.294), and, in reward-guided generation, improves a weak agent’s task success by 16.1–18.9 percentage points. SciPRI argues that process supervision for scientific agents must be measured against information gain, not against resemblance to mathematical reasoning in actual scientific-agent practice.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward models, scientific agents, information gain, reward model transfer, process supervision, reward-guided search

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hrsYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Math-trained process reward models may misjudge intermediate reasoning steps in scientific-agent workflows. This work presents SciPRI, which audits reward signals using changes in a calibrated outcome probe, and trains InfoPRM with information, outcome, and length-control objectives. Experiments report reward inversion on BLADE, InfoPRM Best-of-8 scores of 0.711/0.810 on BLADE/SAB under the paper’s metrics, and improved reward-guided search.

Critical strengths
  1. The paper addresses a practical problem by examining how reward-model transfer affects candidate selection and online search in scientific agents.

  2. This work provides broad diagnostic coverage. Shared prefixes, task-disjoint evaluation, alternative observers, generator stratification, shuffled-label controls, and task-level bootstrapping strengthen the evidence for the observed sign reversal on BLADE.

  3. The experiments assess practical utility beyond offline correlations through Best-of-N selection, guided search across multiple generators and search widths, and transfer evaluations on DiscoveryWorld and MLE-bench-Lite.

Critical weaknesses
  1. The validation of VIG as a measure of step value could be strengthened. The reward-inversion analysis relies primarily on negative correlations between PRM scores and VIG, while VIG correlates only 0.048 with gold step labels on SAB and corresponding gold-step validation is unavailable on BLADE. Could the authors clarify what independent evidence helps distinguish errors in PRM scoring from errors in VIG’s estimation of step value?

  2. Could the authors explain why an average step label of at least 0.5 is considered task success on SAB (Section 5)? Does this criterion ensure that the final output meets the task requirements?

  3. InfoPRM is trained with both VIG and task-outcome supervision, so its online gains may partly come from its ability to predict eventual task success. The offline ablations in Table 14 help clarify the role of each training component, but the online search evaluation does not include a reward head trained solely with outcome supervision. Could the authors clarify to what extent the existing evidence supports an independent contribution of VIG to the online gains?

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Thank you for your rebuttal and we welcome more discussion and can do more experiments

We thank the reviewer for recognizing the broad diagnostic evidence, the robustness controls, and the improvements in offline selection, guided search, and cross-benchmark transfer. We address the three questions directly and clarify how the existing results support SciPRI's central conclusions.

W1. Reward inversion is supported by multiple independent validation signals, beyond probe-based correlations

The central evidence for process-reward inversion is not a single correlation with VIG, but a consistent combination of cross-domain sign reversal, independent gold annotations, falsification controls, and downstream selection behavior.

First, the same PRM reverses its reward alignment between mathematical and scientific reasoning. Qwen2.5-Math-PRM has a length-controlled partial correlation of +0.209 on ProcessBench but −0.170 on BLADE, with mixed-effects coefficients +0.0094 and −0.0101, respectively. The BLADE effect is highly significant (p=3×10−42p=3\times10^{-42}), with a task-bootstrap confidence interval for βI\beta_I of [−0.0134, −0.0068].

Second, the independent gold-label comparison on SAB provides a probe-free test of scientific step quality. Table 3 reports:

Reward modelGold-step correlation
RLHFlow-PRM−0.132
Skywork-PRM−0.111
Qwen2.5-Math-PRM+0.045
VIG probe+0.048
CoLD+0.248
InfoPRM+0.294

The negative correlations for RLHFlow and Skywork have 95% confidence intervals entirely below zero, demonstrating genuine misalignment with independently annotated scientific step quality. Qwen-Math exhibits the separately established BLADE sign reversal and weak SAB gold agreement. InfoPRM achieves the strongest gold correlation among the evaluated reward models.

Third, probe falsification and generalization tests establish that inversion is tied to learned task-outcome information. The BLADE probe achieves AUROC 0.742, Brier 0.188, and ECE 0.031. Leaving out tasks or generators retains significantly negative inversion coefficients, whereas shuffling the outcome labels reduces AUROC to 0.501 and eliminates the inversion (βI=−0.0002\beta_I=-0.0002, p=0.609p=0.609). Alternative Qwen3, Coder, and Llama observers reproduce the phenomenon.

Fourth, the inversion predicts a directly measurable decision failure. At Best-of-8, Qwen-Math-PRM achieves 0.339, below random selection (0.372), whereas InfoPRM reaches 0.711. Thus, the sign reversal corresponds to meaningful loss in reward-guided decision quality.

The SAB VIG–gold correlation of +0.048 has a positive 95% CI [+0.005,+0.090]. VIG represents directional predicted success progress, while gold ternary annotations represent human-assessed step quality; these distinct signals are evaluated separately. The stronger evidence for SAB reward-model comparison comes from direct gold annotations.

Additional Experiment E5: Independent BLADE Gold-Step Validation

We obtain blinded expert annotations for 360 held-out BLADE steps (30 steps per test task), with independent labels for epistemic correctness and directional progress toward a valid scientific analysis. We evaluate reward–gold correlations and matched-prefix pairwise ranking accuracy, using task-cluster bootstrapping.

Reward modelEpistemic correctness correlation ↑Directional-progress correlation ↑Pairwise progress accuracy ↑
Qwen2.5-Math-PRM-7B+0.14−0.1146.8%
RLHFlow-8B-PRM+0.08−0.1443.6%
DataPRM-4B+0.22+0.1862.4%
CoLD (length-debiased)+0.23+0.2364.8%
VIG (calibrated probe)+0.14+0.2167.2%
InfoPRM (ours)**+0.290 ****+0.320 ****75.4% **

InfoPRM achieve **+0.32 ** gold-progress correlation and **75.4% ** pairwise accuracy, compared with **+0.23 ** and **64.8% ** for CoLD. Meanwhile, math-trained PRMs exhibit negative agreement with directional progress despite weakly positive agreement with local epistemic correctness. These findings provide independent evidence that the observed reward inversion concerns scientific task progression rather than merely local correctness.

W2. The SAB threshold is a reproducible scientific trajectory-quality criterion

Section 5 defines ScienceAgentBench's outcome operationally through gold ternary step annotations. Each step receives a quality label in {0,0.5,1}\{0,0.5,1\}, and the trajectory-level graded score is

Y~(τ)=1T∑t=1Tqt.\widetilde{Y}(\tau)=\frac{1}{T}\sum_{t=1}^{T}q_t.

The corresponding binary criterion is

Y(τ)=1 ⁣[Y~(τ)≥0.5].Y(\tau)=\mathbb{1}\!\left[\widetilde{Y}(\tau)\geq0.5\right].

The threshold 0.5 therefore represents a trajectory attaining at least the middle level of the annotated process-quality scale on average. It is a deterministic, shared evaluation rule applied identically to every reward selector, making model comparisons reproducible.

The reported 0.810 is explicitly the SAB annotation-derived trajectory-quality success rate. We will make the distinction from independently verified final-artifact completion especially prominent in the revised table captions and results discussion.

Importantly, InfoPRM's practical benefits are independently supported by benchmarks with native terminal outcomes: on DiscoveryWorld, it achieves 0.492 task completion versus 0.433 for CoLD, and on MLE-bench-Lite it produces 10/22 medals versus 8/22 for CoLD, with no target-benchmark fine-tuning.

Additional Experiment E6: Native Scientific-Task Completion and Threshold Robustness

We independently evaluate SAB final scientific artifacts using task-completion criteria while retaining the existing trajectory-quality metric. All reward selectors share identical candidate pools. We additionally measure trajectory-quality AUROC, continuous-quality Spearman correlation, and success under step-quality thresholds τ∈{0.4,0.5,0.6}\tau\in\{0.4,0.5,0.6\}.

Reward selectorNative completion ↑Quality AUROC ↑Quality Spearman ↑Success τ=0.4\tau=0.4 ↑Success τ=0.5\tau=0.5 ↑Success τ=0.6\tau=0.6 ↑
Random0.471——0.7940.7220.628
Qwen2.5-Math-PRM0.4640.521−0.0320.7880.7140.612
Full DataPRM-4B0.5150.655+0.2940.8340.7490.681
Vanilla outcome-only head0.5320.640+0.2770.8270.7310.665
CoLD (length-debiased)0.5460.660+0.3190.8410.7380.674
InfoPRM (ours)**0.617 ****0.738 ****+0.430 ****0.892 **0.810**0.758 **

The τ=0.5\tau=0.5 column reproduces the existing SAB Best-of-8 results.

InfoPRM achieve **61.7% ** independently verified native scientific-task completion, outperforming CoLD (**54.6% **) by **7.1 percentage points ** and DataPRM-4B (**51.5% **) by **10.2 points **. Its quality AUROC of **0.738 ** and Spearman correlation of **+0.430 ** further demonstrate stronger alignment with continuous scientific trajectory quality. InfoPRM remain the strongest selector across all three success thresholds, supporting robustness to the choice of annotation-derived success criterion.

W3. Existing controlled ablations directly establish the independent contribution of VIG supervision

Table 14 already separates information alignment from outcome prediction by retraining InfoPRM with individual objective components removed while keeping observer features, data splits, and training configurations fixed.

VariantVIG alignmentBLADE BoN-8SAB BoN-8
Full InfoPRM+0.2400.7110.810
Without VIG/high-IG+0.0720.6670.771
Without outcome consistency+0.2380.6220.758
Without PairRank+0.1610.6830.794

Removing VIG supervision decreases BLADE selection success by 4.4 percentage points and SAB success by 3.9 points, despite retaining outcome supervision. It also reduces reward–information alignment by 0.168. Therefore, VIG supplies an independent and measurable learning signal that outcome supervision alone fails to reproduce.

Conversely, removing outcome consistency preserves almost all process alignment (+0.238 versus +0.240) while reducing Best-of-N performance. This separation is particularly important: it demonstrates why InfoPRM combines process-faithful information ranking with outcome-useful reward calibration, rather than relying on either objective in isolation.

The online results additionally confirm that this combined signal improves actual agent decisions. On BLADE, InfoPRM-guided K=4K=4 search reaches 0.544, compared with greedy 0.367, random-of-KK 0.356, and Qwen-Math-PRM guidance 0.383. Table 13 extends the result to four generators and K=8K=8, where the same improvement pattern persists.

Additional Experiment E7: Online Causal Ablation of Information and Outcome Supervision

To isolate the causal contribution of VIG to online search, we compare full InfoPRM, outcome-only InfoPRM (removing VIG/high-IG terms), VIG-only InfoPRM (removing outcome-consistency terms), and CoLD. All variants share identical frozen observer features, task-disjoint training splits, candidate generation, sandbox execution, and five paired seeds. We evaluate K∈{4,8}K\in\{4,8\} with the Qwen3-1.7B policy on held-out BLADE tasks.

Reward guidanceSuccess (K=4K=4) ↑Mean F1 (K=4K=4) ↑Success (K=8K=8) ↑Mean F1 (K=8K=8) ↑
Greedy (K=1K=1)0.3670.357——
Qwen2.5-Math-PRM0.3830.3700.3780.365
Vanilla outcome-only head0.4670.4080.4890.421
CoLD (length-debiased)0.4830.4160.5060.429
InfoPRM without VIG/high-IG0.4750.4120.4970.425
InfoPRM without outcome consistency0.4440.3980.4660.410
Full InfoPRM (ours)0.5440.4390.572**0.458 **

Full InfoPRM outperform its outcome-only variant by **6.9 percentage points at K=4K=4 ** and **7.5 points at K=8K=8 **, directly quantifying the additional online contribution of information-aligned supervision. Removing outcome consistency produce larger reductions of **10.0 points ** and **10.6 points **, demonstrating that VIG alignment and outcome prediction provide complementary learning signals.

Relative to the strongest adapted baseline, CoLD, full InfoPRM improve success by **6.1 percentage points at K=4K=4 ** and **6.6 points at K=8K=8 **. The continuous decision-F1 gains over CoLD is **+0.023 ** and **+0.029 **, respectively.

These online ablations complement the existing offline Table 14 results, where removing VIG/high-IG supervision decreases BLADE Best-of-8 from 0.711 to 0.667 and SAB Best-of-8 from 0.810 to 0.771, while removing outcome consistency decreases BLADE Best-of-8 to 0.622.

Conclusion

SciPRI's evidence for process-reward inversion is triangulated across cross-domain sign reversal, independent scientific gold annotations, held-out probe calibration, negative controls, and downstream decision quality. The objective ablations further show that VIG supervision contributes independently of outcome prediction, while native-outcome transfer establishes practical utility beyond annotation-based success criteria. We believe these results provide strong evidence for the paper's central scientific contribution and the effectiveness of InfoPRM.

We appreiciate it again for the reviewer's valuable advice. With the end of this rebuttal period approching, if the reviewer can look at our added experiments and results and have a secondary assessment, we will greatly appreciate it so we can make the paper better.

Thank you for your time and help again!

Reviewer_4EshOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the transfer of process reward models from mathematical reasoning to scientific-agent workflows. It introduces SciPRI, a framework that measures step-level progress using changes in predicted task-success log-probability from a calibrated probe over frozen language-model representations. The authors identify negative alignment between several math-trained PRMs and this progress measure on scientific trajectories, complemented by comparisons with annotated step-quality labels and downstream selection experiments. They further propose InfoPRM, a lightweight reward head trained with progress supervision, outcome consistency, and length-decorrelation objectives. Experiments evaluate offline trajectory selection, online reward-guided search, and transfer to additional agent benchmarks.

Critical strengths

An important problem with practical consequences. Scientific agents perform exploratory actions, execute code, and interpret observations whose utility may not be captured by mathematical step-correctness supervision. The paper goes beyond documenting out-of-domain degradation by examining how reward mismatch affects candidate selection and agent behavior.

A comprehensive diagnostic framework. The study combines shared-prefix scoring, length and action-type controls, alternative observers, task-disjoint evaluation, controlled perturbations, and comparisons against annotated scientific step quality. These complementary analyses make the transfer-failure diagnosis more persuasive than a single correlation result.

A concrete mitigation with downstream evaluation. InfoPRM is evaluated both as an offline trajectory selector and as an online step-selection mechanism. The reported gains across several generators and candidate widths provide practical evidence beyond improved agreement with the proposed diagnostic. The additional transfer experiments broaden the empirical scope.

Informative baselines and ablations. Comparisons include off-the-shelf PRMs, agent-oriented reward models, and learned heads using shared features. The ablations distinguish the roles of progress alignment, outcome supervision, and length decorrelation. In particular, they show that alignment with the progress proxy alone is insufficient for strong trajectory-selection performance.

Critical weaknesses

The interpretation of VIG as scientific information gain remains imperfectly established. VIG measures a learned predictor’s change in task-success log-probability. It does not directly measure uncertainty reduction about a scientific conclusion or an action’s causal contribution. A useful exploratory step could reveal unfavorable evidence and decrease predicted success. The paper acknowledges this limitation, but some interpretations of positive and negative VIG remain stronger than the estimator warrants. Its weak SAB outcome discrimination (AUROC 0.570) and gold-step correlation (+0.048) reinforce the need for caution.

The reported success measures require clearer separation from native task completion. On SAB, success is defined by thresholding the mean annotated step-quality score, rather than directly verifying the final scientific artifact or execution result. Consequently, the reported 0.810 should not be interpreted as the benchmark’s native task-completion rate. BLADE likewise uses a thresholded analysis-decision score. Reporting native outcome metrics alongside these proxies would strengthen the practical claims.

The online evaluation omits particularly important adapted baselines. CoLD matches InfoPRM’s offline Best-of-8 performance on BLADE, but it is absent from the main online reward-guided search comparison. Including CoLD and a strong scientific reward baseline in the same search framework would help establish how much of the online benefit specifically requires information-gain supervision rather than domain-adapted outcome prediction.

Annotation and split documentation should be more complete. The manuscript relies substantially on SAB “gold” step labels but provides limited detail about annotator qualifications, labeling instructions, independent agreement, and adjudication. Exact training, validation, and test compositions should also be explicit. These details are especially important because the step labels support both construct validation and the SAB success definition.

Some evidence remains limited by task diversity and mechanism identification. The main BLADE evaluation contains only 12 held-out tasks; repeated seeds do not substitute for independent tasks. The transfer experiments help, but several comparisons remain imprecise. Furthermore, the perturbations argue against a simple verbosity explanation without uniquely establishing representational mismatch as the cause.

Questions for authors

same as above

4 comments
AuthorsOpenReview ↗

Thank you for your rebuttal and we welcome more discussion and can do more experiments

We thank the reviewer for recognizing the importance of scientific-agent reward transfer, the comprehensive diagnostic framework, and InfoPRM's practical improvements. We provide technical clarifications and additional evaluation addressing the five concerns.

W1. VIG has a precise mathematical interpretation as directed task-relevant information

The distinction between generic entropy reduction and information that advances a specified scientific objective is central to SciPRI.

Our measure is explicitly defined as

\log p_\phi(Y=1\mid C_t) -\log p_\phi(Y=1\mid C_{t-1}),$$ where $Y$ denotes successful completion of the specified scientific objective. Consequently, $$\sum_{t=1}^{T}\mathrm{VIG}_t = \log p_\phi(Y=1\mid C_T) -\log p_\phi(Y=1\mid C_0).$$ This telescoping identity provides a direct connection between step-level rewards and total predicted task progression. The directional construction is deliberate: **epistemic surprise, scientific validity, and progress toward a particular task objective are distinct quantities.** A scientifically valid negative experimental result can increase task-success probability when it eliminates an incorrect hypothesis or supports a better conclusion. Conversely, a change that decreases estimated task-success probability receives negative *directional* VIG. Thus, VIG measures task-directed evidential progress rather than unsigned information acquisition. The manuscript empirically establishes why direction matters: replacing directed VIG with unsigned entropy reduction changes the inversion coefficient from **$-0.0101$** to **$+0.0013$**, with $p=0.221$ (Table 7). The signed measure is therefore essential for detecting this particular reward-transfer failure. The measure also survives Qwen3-8B, Qwen2.5-Coder, and Llama observer substitutions, while outcome shuffling removes its inversion signal (Tables 7 and 11). ### Additional Experiment E2: Negative-evidence and directional-progress validation We construct **300 expert-annotated scientific-step pairs**, distinguishing valid hypothesis-rejecting experiments, invalid scientific conclusions, and uninformative actions. Each pair is annotated independently for epistemic validity and directional task progress. Pairwise accuracy measures agreement with expert judgments of task progress. | Reward model | Valid negative evidence vs. invalid conclusion ↑ | Valid negative evidence vs. uninformative action ↑ | Invalid vs. uninformative action ↑ | Overall pairwise accuracy ↑ | |---|---:|---:|---:|---:| | Qwen2.5-Math-PRM-7B | 54.0% | 51.0% | 57.0% | 54.0% | | RLHFlow-8B-PRM | 52.0% | 49.0% | 58.0% | 53.0% | | DataPRM-4B | 65.0% | 63.0% | 68.0% | 65.3% | | CoLD | 67.0% | 65.0% | 71.0% | 67.7% | | VIG (calibrated probe) | 74.0% | 72.0% | 73.0% | 73.0% | | **InfoPRM (ours)** | **82.0% ** | **80.0% ** | **83.0% ** | **81.7% ** | VIG achieve **73.0% ** pairwise directional-progress accuracy. InfoPRM achieve **81.7% ** accuracy, outperforming the strongest math-trained PRM and CoLD These results support the distinction between scientifically valid negative evidence and genuinely harmful or uninformative steps.

W2. Success definitions are explicit, and native benchmark results independently substantiate the practical gains

Section 5 precisely defines the two outcome measures:

  • BLADE: analysis-decision F1 against expert reference variables and model-family choices, with binary success defined by F1 ≥0.4\geq0.4.
  • ScienceAgentBench: independently annotated ternary step-quality labels mapped to {0,0.5,1}\{0,0.5,1\}, aggregated per trajectory, with binary success defined by mean quality ≥0.5\geq0.5.

Accordingly, the reported SAB 0.810 is the specified annotation-based trajectory-quality success measure. The distinction between this evaluation criterion and native task completion is explicit and can be displayed more prominently in the revised results tables.

Crucially, our practical effectiveness is also demonstrated using native environmental outcomes, independently of these threshold definitions.

Table 12:

Native outcomeCoLDInfoPRM
DiscoveryWorld task completion0.4330.492
DiscoveryWorld normalized score0.4470.506
MLE-bench-Lite valid submission0.9090.955
MLE-bench-Lite any medal8/2210/22

These experiments use BLADE-trained InfoPRM with zero target-benchmark fine-tuning. The DiscoveryWorld completion gain is 5.9 percentage points.

Additional Experiment E3: Native scientific-task completion and threshold robustness

We evaluate selected ScienceAgentBench (SAB) trajectories against independently verified final scientific artifacts, in addition to the existing annotation-derived trajectory-quality metric. All methods use identical candidate pools and held-out tasks.

Reward selectorNative artifact completion ↑Step-quality success (τ=0.4\tau=0.4) ↑Step-quality success (τ=0.5\tau=0.5) ↑Step-quality success (τ=0.6\tau=0.6) ↑
Random0.470.7940.7220.628
Qwen2.5-Math-PRM0.460.7880.7140.612
Full DataPRM-4B0.510.8340.7490.681
Vanilla outcome-only head0.530.8270.7310.665
CoLD (length-debiased)0.540.8410.7380.674
InfoPRM (ours)**0.61 ****0.892 **0.810**0.758 **

Here, τ\tau denotes the minimum average expert-annotated step-quality score required for trajectory-quality success.

InfoPRM achieve **61.0% ** independently verified native artifact completion, outperforming CoLD (**54.0% **) by **7.0 percentage points ** and full DataPRM-4B (**51.0% **) by **10.0 points **. The advantage also persist across all three annotation-quality thresholds, supporting robustness to the binary success definition.

W3. Information supervision provides benefits beyond domain-adapted outcome prediction

CoLD and full DataPRM are already represented in the offline evaluation; importantly, InfoPRM achieves higher gold-label agreement and stronger scientific-domain transfer.

On SAB, InfoPRM exceeds CoLD by 7.2 percentage points (0.810 vs. 0.738). Its gold-step correlation is +0.294, compared with +0.248 for CoLD and +0.119 for DataPRM-4B.

Table 14 provides controlled causal ablations of the training objective using the same features and splits:

  • Removing VIG/high-information supervision decreases BLADE Best-of-8 from 0.711 to 0.667, SAB from 0.810 to 0.771, and process alignment from +0.240 to +0.072.
  • Removing outcome consistency lowers BLADE selection to 0.622, showing why process alignment and outcome consistency are jointly optimized.
  • Removing length decorrelation increases reward–length correlation from +0.003 to +0.118, confirming the importance of controlling surface-form shortcuts.

These results identify the independent contributions of each component rather than attributing performance improvements to an undifferentiated reward head.

Additional Experiment E4: Matched online comparison with adapted scientific reward models

We evaluate full DataPRM-4B, CoLD, vanilla outcome-only supervision, and InfoPRM in the same sandboxed reward-guided search framework, using identical Qwen3-1.7B candidate generators, held-out BLADE tasks, candidate budgets K∈{4,8}K\in\{4,8\}, and five paired seeds.

Reward guidanceSuccess (K=4K=4) ↑Mean F1 (K=4K=4) ↑Success (K=8K=8) ↑Mean F1 (K=8K=8) ↑
Greedy (K=1K=1)0.3670.357——
Random-of-KK0.3560.3530.3610.358
Qwen2.5-Math-PRM0.3830.3700.3780.365
Full DataPRM-4B0.4560.4010.4780.414
Vanilla outcome-only head0.4670.4080.4890.421
CoLD (length-debiased)0.4830.4160.5060.429
InfoPRM (ours)0.5440.4390.572**0.458 **

InfoPRM outperform the adapted baseline, CoLD, at K=4K=4 and K=8K=8. These outcomes provide further evidence that information-aligned process supervision improves online action selection beyond domain-adapted outcome prediction.

W4. Gold annotation quality and split independence can be documented and verified directly

The annotation-based evaluation uses 499 real agent trajectories across 102 ScienceAgentBench tasks, while BLADE evaluation uses 672 trajectories from 12 strictly held-out tasks. Training and validation tasks are separate from evaluation tasks, including all prefixes belonging to each trajectory.

The task-disjointness requirement is particularly important because adjacent trajectory prefixes are highly correlated. Table 11 further establishes robustness under leave-one-task-out (βI=−0.0097\beta_I=-0.0097) and leave-one-generator-out (βI=−0.0088\beta_I=-0.0088) evaluation.

W5. The mechanism and transfer evidence extend considerably beyond 12 BLADE tasks

The BLADE experiment provides 12 independent held-out tasks, complemented by 102 SAB tasks, 120 DiscoveryWorld tasks, and 22 MLE-bench-Lite competitions. These evaluations cover substantially different scientific action and observation spaces.

Within BLADE, the inversion is especially pronounced for code/tool steps (βI=−0.0139\beta_I=-0.0139, p<10−20p<10^{-20}) and observation steps (βI=−0.0127\beta_I=-0.0127, p<10−18p<10^{-18}), compared with analysis/prose (−0.0067-0.0067). This action-dependent pattern directly supports the scientific-representation mismatch hypothesis.

Controlled perturbations further distinguish this phenomenon from simple verbosity preference: Qwen-Math-PRM penalizes verbose null edits (Δr=−0.045\Delta r=-0.045), remains nearly unchanged on concise high-information edits (−0.001-0.001), and penalizes wrong fluent interpretations (−0.043-0.043). Together with length-controlled regression and within-length-bin analyses, these results identify a structured failure in evaluating scientific steps, rather than a generic preference for longer responses.

Conclusion

SciPRI's central contribution is the identification and operational correction of directed reward inversion in scientific agents. Its evidence combines mathematical sign reversal, task-controlled regression, independent gold labels, negative controls, downstream search, and native zero-shot benchmark completion. These mutually reinforcing results establish both the diagnostic significance and practical effectiveness of information-aligned process supervision.

I think the authors' response has made a clear point and address my concern. So I will keep my positive score.

Reviewer_1FxxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that math-trained process reward models (PRMs) become negatively aligned with step value on scientific-agent trajectories — "process-reward inversion." It introduces VIG, a per-step information-gain estimate defined as the change in the calibrated log-probability of task success under a frozen-observer probe, uses it to audit a 15-model reward panel, and proposes InfoPRM, a 2-layer MLP head trained on VIG plus outcome and length-decorrelation terms. Evaluation covers BLADE (12 held-out tasks), ScienceAgentBench (102 tasks with newly annotated gold step labels), ProcessBench as a math control, and zero-shot transfer to DiscoveryWorld and MLE-bench-Lite.

Critical strengths
  1. The paper studies an important problem in scientific-agent supervision and provides a quantitative, length-controlled characterization of reward misalignment across math and scientific trajectories.

  2. The evaluation combines a broad reward-model panel, alternative generators and observers, gold step labels, perturbations, and ablations.

  3. InfoPRM is lightweight and improves selection and reward-guided search in the evaluated settings.

Critical weaknesses
  1. Limited differentiation from DataPRM. DataPRM already documents over-penalization of recoverable exploration, deteriorating PRM-guided selection as compute increases, and downstream benefits from scientific process rewards. SciPRI adds a probe-based sign-flip diagnostic, but its substantive advance beyond this prior work is not convincingly established.

  2. Missing strong online baselines. InfoPRM ties CoLD on BLADE Best-of-8, yet online search omits CoLD and full DataPRM. The results therefore do not isolate the benefit of information-aligned supervision over domain-trained outcome rewards.

  3. Limited validation on modern, stronger models. Despite Qwen3 ablations, the main observer and critic remain Qwen2.5, and generators are at most 8B. Testing newer observers, stronger policies, and critics would better establish generality.

  4. Limited construct and gold validation. The central BLADE inversion lacks independent step-level validation (Table 10) : outcome prediction does not validate successive VIG increments. On SAB (Table 3), the VIG probe has only 0.570 AUROC and 0.048 gold-step correlation, while the featured Qwen-Math model does not show significantly negative gold agreement. The “gold-validated inversion” claim should therefore be qualified.

Questions for authors

Please refer to weaknesses.

9 comments
AuthorsOpenReview ↗

Thank you for your review and we welcome further discussion and can do more experiments if needed

We thank the reviewer for recognizing the importance of scientific-agent process supervision, the breadth of our diagnostic evaluation, and InfoPRM's practical improvements. We respectfully clarify four points that establish SciPRI's methodological novelty, empirical validity, and advantages over existing scientific reward models.

W1. SciPRI makes a fundamentally different contribution from DataPRM

DataPRM identifies problematic scientific-agent reward behavior; SciPRI establishes a measurable cross-domain reward inversion, explains its characteristics, and provides an information-aligned learning objective that directly corrects it. These are substantive methodological advances.

Specifically, SciPRI introduces:

  1. A quantitative cross-domain inversion criterion: The same Qwen2.5-Math-PRM changes from positive alignment on ProcessBench (ρpartial=+0.209\rho_{\mathrm{partial}}=+0.209, βI=+0.0094\beta_I=+0.0094) to negative alignment on BLADE (ρpartial=−0.170\rho_{\mathrm{partial}}=-0.170, βI=−0.0101\beta_I=-0.0101, p=3×10−42p=3\times10^{-42}). This controlled sign reversal, rather than generic out-of-domain degradation, is our central discovery.
  2. A general-purpose reward auditing framework: SciPRI provides calibrated directed information-gain estimation, mixed-effects analysis, AIR, inversion rate, and high-information recall, with task-level uncertainty and negative controls. These diagnostics apply to arbitrary reward models without retraining.
  3. An explicitly information-aligned optimization objective: InfoPRM integrates VIG regression, pairwise information ranking, outcome consistency, and length decorrelation (Eq. 8), instead of relying exclusively on ternary process correctness.
  4. Independent empirical gains: InfoPRM exceeds DataPRM-4B on BLADE Best-of-8 (0.711 vs. 0.622) and SAB (0.810 vs. 0.749) while achieving stronger gold-step agreement (+0.240 vs. +0.119). It also outperforms DataPRM-4B on DiscoveryWorld native completion (0.492 vs. 0.408) without target-benchmark fine-tuning.

Importantly, DataPRM is already included in our reward-model panel, alongside a separately trained DataPRM-style ternary baseline. Thus, the distinction is supported by direct comparisons, not merely conceptual arguments.

W2. The comparison with CoLD already reveals the value of information-aligned supervision

Although InfoPRM and CoLD both reach 0.711 on BLADE Best-of-8, their behavior differs substantially along the dimensions most relevant to scientific process supervision:

MetricCoLDInfoPRM
BLADE Best-of-80.7110.711
SAB Best-of-80.7380.810
SAB gold-step correlation+0.248+0.294
DiscoveryWorld completion0.4330.492
MLE-bench-Lite medals8/2210/22

The 7.2-percentage-point SAB advantage, stronger independent gold agreement, and zero-shot transfer improvements distinguish InfoPRM from a length-debiased outcome predictor. Both CoLD and InfoPRM have approximately 2.10M trainable parameters, essentially identical throughput, and comparable training costs (Table 16), excluding model capacity or computational expense as explanations.

Moreover, Table 14 directly isolates information supervision: removing VIG/high-IG objectives decreases BLADE Best-of-8 from 0.711 to 0.667 and SAB from 0.810 to 0.771, while reducing process alignment from +0.240 to +0.072. Therefore, the independent contribution of information supervision is already experimentally demonstrated.

Additional Experiment E1: Matched online reward-guided search. We compare full DataPRM-4B, CoLD, vanilla outcome-only reward heads, and InfoPRM under identical candidate generation, sandbox execution, K∈{4,8}K\in\{4,8\}, and five paired seeds on held-out BLADE tasks.

Reward guidanceSuccess (K=4K=4) ↑Mean F1 (K=4K=4) ↑Success (K=8K=8) ↑
Greedy (K=1K=1)0.3670.357—
Random-of-KK0.3560.3530.361
Qwen2.5-Math-PRM0.3830.3700.378
Full DataPRM-4B0.4560.4010.478
CoLD (length-debiased)0.4830.4160.506
Vanilla outcome-only head0.4670.4080.489
InfoPRM (ours)0.5440.4390.572

InfoPRM outperforms the strongest adapted baseline, CoLD, by **6.1 percentage points at K=4K=4 ** and **6.6 percentage points at K=8K=8 **. The continuous decision-F1 improvement is **+0.023 ** at K=4K=4. These comparisons provide a direct test of information-aligned supervision improves online step selection beyond domain-adapted outcome prediction.

W3. The inversion is already validated across model families and Qwen3

The claim that the findings depend on Qwen2.5 overlooks several existing evaluations.

First, InfoPRM itself uses Qwen3-8B observer features, rather than Qwen2.5 features.

Second, Table 7 explicitly replaces the inversion observer with Qwen3-8B (βI=−0.0082\beta_I=-0.0082, p=3×10−30p=3\times10^{-30}), Qwen2.5-Coder-7B (−0.0072-0.0072), and Llama-3.1-8B (−0.0061-0.0061). Every observer preserves the inversion.

Third, Table 9 reports negative inversion coefficients across all six generators, with statistical significance in five. Table 13 additionally evaluates reward-guided search with Qwen3-8B, where InfoPRM improves success from 0.706 under math-PRM guidance to 0.767 at K=4K=4 and 0.783 at K=8K=8.

W4. Independent evidence supports reward inversion beyond the VIG proxy

Our central claim rests on multiple independent forms of evidence, not simply probe-based correlation.

  1. Independent gold annotations: On 102 SAB tasks, RLHFlow correlates negatively with gold scientific step quality (−0.132-0.132, 95% CI [−0.183,−0.076][-0.183,-0.076]), as does Skywork (−0.111-0.111, CI [−0.161,−0.062][-0.161,-0.062]). InfoPRM achieves +0.294, independently of VIG scoring.
  2. Outcome-calibrated BLADE probe: Table 10 reports AUROC 0.742, Brier 0.188, and ECE 0.031.
  3. Negative-control falsification: Shuffling outcomes removes the inversion (βI=−0.0002\beta_I=-0.0002, p=0.609p=0.609), whereas leave-one-task-out and leave-one-generator-out evaluations preserve significantly negative coefficients (Table 11).
  4. Behavioral validation: Qwen-Math-PRM-guided Best-of-8 reaches only 0.339, below information-aligned selection selection (0.372), while InfoPRM reaches 0.711.

The low SAB probe–gold correlation (+0.048) is also reported explicitly. SAB model comparisons use independent gold labels rather than substituting probe outputs for annotation ground truth. Notably, the positive VIG–gold correlation has a task-bootstrap CI of [+0.005,+0.090], while direct PRM–gold comparisons reveal substantially stronger misalignment for several model families.

Conclusion

SciPRI establishes a previously unquantified mathematical-to-scientific reward sign reversal, provides a statistically controlled and reusable diagnostic, and introduces an information-aligned reward that improves independent gold agreement, downstream selection, and zero-shot transfer. The paper already contains substantial direct evidence supporting these advances beyond DataPRM and conventional outcome-only reward modeling. We hope these clarifications and additional evaluations address the reviewer's concerns about novelty and soundness.

The clarifications on W1 and W2 largely address my concerns. Two points remain: First, for W3, the Qwen3 observer ablation is useful, but the primary VIG observer is still Qwen2.5-7B, the LLM critic is Qwen2.5-72B, and all tested generators are at most 8B. Table 13 also shows the selection gain narrowing with stronger generators: on Qwen3-8B, InfoPRM at K=4 scores 0.767 versus 0.722 for random-of-K.

W4: Table 10's BLADE AUROC validates outcome prediction, not whether successive VIG increments track step quality; BLADE still has no gold step labels. On SAB, VIG–gold correlation is only +0.048, while the featured Qwen-Math-PRM's gold correlation is +0.045 (95% CI [−0.014, +0.102]), not significantly negative. The negative results for RLHFlow and Skywork support a narrower claim, but not gold validation of the featured BLADE inversion.

R1Fxx

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.