acceptodds
Withdrawn from ICLR 2027

CrossView-PRM: Falsifiable Step-Level Credit from Correlated Noisy Views

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning for long-horizon language agents commonly requires verifiable outcomes or externally supervised process reward models (PRMs), assumptions that fail in open-ended planning with multiple valid solutions and correlated model judges. We introduce CrossView-PRM, a target-verifier-free reward-construction framework, with three main contributions. Rather than treating agreement as correctness, the framework uses unlabeled target statistics only to detect instability and relies on source diagnostic domains to test whether learned reliability rules transport. Matched-budget sibling actions turn each decision into a controlled counterfactual comparison, yielding a non-telescoping ordinal reward without target-domain labels. Semantic, model, and temporal views are residualized and redundancy-weighted, while a source-calibrated gate abstains under view collapse, disagreement, or domain shift; a conditional analysis bounds accepted reward-sign error. Controlled causal planning, TravelPlanner, and TripScore jointly test reward validity, transport, policy utility, robustness, and cost. CrossView-PRM reaches \(0.45\) Kendall's \(\tau\), \(76%\) pairwise accuracy, and \(30%\) TravelPlanner Final Pass, improving over Majority-view PRM by \(0.12\) \(\tau\) and \(8.5\) final-pass points. Shared-bias stress tests and resource accounting further characterize the reward's reliability and cost.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Process Reward Model, Long-horizon Language Agent

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_WLLyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CrossView-PRM, a target-verifier-free reward-construction framework for long-horizon language agents. It compares executed actions with sibling actions from the same decision state under matched budgets. Semantic, model, and temporal views produce soft preferences over these action pairs, which are aggregated using correlation-aware weighting and screened by a source-calibrated gate. Accepted labels are used to train a prefix potential, which is then frozen. The primary reward is obtained by comparing the executed action with its siblings using the trained prefix potential, supplemented by potential-difference shaping. The policy is then updated using GRPO. The method is compared with other PRMs and self-rewarding baselines, with outcome-only GRPO included as a supervision-assisted reference.

Critical strengths

The paper explicitly addresses the concern that agreement among multiple judges does not necessarily reflect true action quality, particularly when their errors are correlated. Correlation-aware aggregation and screening by a source-calibrated gate are reasonable mechanisms to address this concern.

Critical weaknesses
  1. The paper is poorly written and difficult to follow. It introduces terms such as ordinal sibling advantage, reliability rule, and fail-closed before clearly explaining their meaning in the method. This makes it difficult to understand the overall approach and assess its technical contributions.
  2. The method relies on a lot of hyperparameters and rules. Although the appendix describes source calibration and hyperparameter search ranges, it remains unclear how sensitive final performance is to these choices and how much recalibration is required for a new target domain. This raises concerns about the practical complexity and generalizability of the approach.
  3. The computational cost is substantial: CrossView-PRM uses 7.5× relative tokens and 2,500 H100 GPU-hours per seed, compared with 4.5× and 1,850 GPU-hours for Majority-view PRM. Although the paper acknowledges this trade-off, the comparisons include only the policy-update budget, not total computation. It therefore remains unclear whether the performance gains justify the additional cost, or whether simpler baselines could achieve comparable results with the same total compute budget.
Questions for authors
  1. Since labeled source diagnostics are available, how does the proposed method compare with directly training a reward model on the same source labels and transferring it to the target domain?
  2. Are the same source calibration data and rules used across all downstream tasks and benchmarks? Which components, if any, require recalibration for a new target domain?
  3. For the ablation study in Appendix E.6, could the authors clarify what homogeneous views means and how online potential is implemented?
  4. Does the method’s performance depend on how well the judges complement each other? Do the authors have any suggestions for selecting judges with complementary views?
Reviewer_f9i8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CrossView-PRM constructs process rewards by comparing an executed action with four alternatives from the same restored decision state, using matched continuation budgets. Seven frozen semantic, model, and temporal views provide preferences. Source-calibrated filtering, residual-correlation weighting, and abstention select comparisons for training a scalar potential. During alternating GRPO updates, a frozen copy provides an ordinal sibling reward and auxiliary shaping. The analysis gives a conditional pair-sign error bound using Cantelli's inequality. Experiments report improvements in synthetic causal-planning reward validity, TravelPlanner final pass rate, and TripScore utility. The paper distinguishes reward construction without a target verifier from evaluator-assisted checkpoint selection, and reports higher resource use than the main baseline.

Critical strengths

I think comparing alternative actions from the same starting state is a sensible choice. It makes the local comparison easier to interpret because the alternatives share the same context and planning budget. I also find the treatment of correlated judge errors useful. The method separates disagreement on an individual comparison from redundancy across views, and uses source problems to calibrate its reliability checks. The ablations that remove matching, redundancy weighting, or abstention support the importance of these components in the tested setting.

The evaluation checks both the quality of the learned rewards and whether they help the agent perform better. TravelPlanner improves by 8.5 percentage points over Majority-view PRM, with improvements in all three training runs. The authors also report the stronger outcome-supervised result and acknowledge that checkpoint selection uses an evaluator and that their method requires more tokens and GPU time. This makes the benefits and limitations easier to assess.

Critical weaknesses

My general concern is whether the evidence is strong enough to justify trusting the learned rewards. The held-out CCP results and the improvements on TravelPlanner and TripScore provide useful to know that the method transfers in practice. But the scope of the reliability guarantee should be clarified. The audit estimates the quantities in the bound on labeled source problems, and I would like the authors to clarify whether the bound is intended as a source-domain diagnostic or also as a guarantee for accepted target comparisons. If the latter, the assumptions connecting the source and target error rates need to be made explicit.

I’m also unsure about a step in the proof. They first bound the error when one comparison passes the filter, then use those bounds when all four comparisons pass. The error rate can change when we keep only that smaller group. Before adding the individual bounds together, the authors need to explain why they still hold for that group. For example, a comparison could be wrong in 10 out of 100 accepted cases. Requiring all four comparisons to pass might leave only 20 cases, including those same 10 mistakes, raising the error rate from 10% to 50%. The original bound therefore needs justification for this smaller group. Also, a comparison reward of +0.5 could become −0.3 after adding shaping and costs, so the authors should clarify which reward their guarantee covers.

Also, the writing of the paper could be enhanced; for example, I had difficulty when the authors introduced anchor abruptly, and I would also like clearer definitions of R-hat and R-star. It is unclear whether they refer only to the comparison reward or to the full reward, including shaping and action costs.

I also found the submitted evidence insufficiently specified for reproducing the audit. The main text promises complete window-level statistics, model revisions, and implementation details, but Appendix E.1 gives three seed summaries and Appendix F describes a future release contract. More details should have been reported. The selected thresholds, frozen auditor serialization and prompts, and a sufficiently detailed CCP generator are not supplied there. Without them, it is difficult to reproduce the mechanism tests or determine exactly when the procedure abstains or stops.

Other limitations: -The method requires an environment that supports branching from the same state. Its comparisons rely on restoring the state, trying alternatives, and obtaining four valid sibling actions. That is feasible in a suitable simulator or planning environment. It can be harder for an agent interacting with a live system where actions have irreversible consequences. -I’m not yet convinced that the gains justify the extra computation. CrossView uses 2,500 H100 GPU-hours per seed, compared with 1,850 for Majority-view PRM, and also uses more tokens.

Questions for authors
  1. Please use a consistent conditioning event for the pair and set bounds in Proposition 1 and Appendix C, define R-hat/R-star, and state the assumption that transfers source moment or error bounds to accepted target comparisons. Which parts are tested source-domain diagnostics, and which are intended as target reliability guarantees?

  2. Can you provide the selected calibration thresholds, exact model versions, and audit results for each training window, including confidence limits and counts of accepted pairs and complete sibling sets? The three seed summaries do not let me check the individual windows. For seed 17 in Table 1 of Appendix E.1, substituting the displayed values gives (0.028/(0.028+0.31^2)+0.022 \approx 0.248), while the reported bound is 0.21. Was this obtained by averaging separately calculated window-level bounds? Please explain

  3. Can you provide the exact auditor prompts, input format, parsing rules, and CCP-generation specifications? I also want to understand how the reference labels are determined when action values are estimated using 256 continuations. If two actions have nearly equal estimated values, how is uncertainty handled before deciding which action is better?

4- Can you compare CrossView with Majority-view PRM under the same total token budget?

Reviewer_zUZQOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1(−1)
Soundness
1
Read review
Paper summary

This paper proposes step-level rewards without target-domain verifiers through counterfactual comparisons. The authors report reward alignment and planning performance on CCP, TravelPlanner, and TripScore.

Critical strengths

The matched-sibling formulation gives step-level credit a well-defined comparison. The authors evaluate alternative actions from the same decision state under matched continuation budgets. Ablations support the importance of matching and abstention, while improvements are presents in both reward alignment and downstream planning.

Critical weaknesses
  1. The figures of this manuscript is likely generated by AI, espically for Fgiure 2. The authors should provide the main components of the proposed method instead of using detailed information. Furthermore, there are some incosisent decsription in Figure, including font size and color.
  2. Expreimental results are not convincing. First, the authors should evalute more benchmarks. Second, the comparison models are not representative. Recent two-year models including Qwen 3.8 should be compared.
  3. The limitted references are concluded. The authors should review related works.
  4. Appendix E.1 is not provided as claimed.
Questions for authors

see above

Reviewer_bcv8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrossView-PRM, a framework for constructing step-level rewards for language agents without target-domain training verifiers. It compares matched sibling actions and aggregates preferences from multiple model-based views using correlation-aware weighting and selective abstention. The accepted preferences are used to train a potential model that provides rewards for GRPO-based policy optimization. Reliability criteria are calibrated on controlled source domains. Experiments on CCP, TravelPlanner, and TripScore report improvements in reward ranking and downstream planning performance, accompanied by ablation studies, robustness evaluations, and computational cost analysis.

Critical strengths
  1. The matched sibling design provides a reasonable way to compare alternative actions from the same decision state while controlling continuation budgets. This makes the resulting preferences more interpretable than comparisons between unrelated trajectory prefixes. The corresponding ablation also shows a substantial performance impact.
  2. The paper evaluates both reward quality and downstream planning performance. The inclusion of multiple seeds, statistical tests, component ablations, and robustness analysis strengthens the empirical evidence. I also appreciate that the authors explicitly acknowledge the additional reward-construction cost rather than claiming equal-compute efficiency.
Critical weaknesses
  1. Removing abstention reduces TravelPlanner Final Pass from 30% to 22%, suggesting that reward filtering contributes substantially to the improvement. However, the simple-mean ablation has different coverage (0.74 vs. 0.68), making it unclear whether the gain comes from correlation-aware modeling or simply selecting more reliable training samples. A coverage-matched comparison with simple confidence-based filtering is needed. Moreover, CrossView requires substantially more computation than Majority-view PRM (7.5× vs. 4.5× tokens), without a total-compute-matched comparison.
  2. Correlation among evaluator predictions does not necessarily reflect correlated errors. In particular, the method may fail when multiple evaluators confidently agree on an incorrect preference. The shared-bias experiments show improved robustness but also substantially reduced coverage, making it difficult to distinguish error detection from conservative filtering. Furthermore, reliability is mainly validated within the controlled CCP domains, while TravelPlanner and TripScore only demonstrate downstream performance gains. It remains unclear whether the claimed reliability transfers to these target domains.
Questions for authors
  1. Does CrossView still outperform simple confidence-based filtering or a strong single judge when coverage and evaluation budgets are matched?
  2. How is the shared-bias intervention constructed? Can CrossView detect cases where all evaluators confidently agree on an incorrect preference, rather than merely rejecting uncertain samples?
  3. Since all four sibling comparisons must pass the gate, what proportion of training anchors actually receives nonzero rewards, and how does this change during training?
  4. What evidence supports the transfer of reliability calibrated on CCP to TravelPlanner and TripScore, beyond improvements in downstream task performance?
  5. Are CCP Kendall's τ\tau and pairwise accuracy evaluated on the same held-out pairs for all methods, or only on each method's accepted samples? If the latter, how are the comparisons made fair?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.