acceptodds
Desk rejected at ICLR 2027

ForkPRM: Learning Embodied Process Verifiers from Executable Counterfactuals

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Process supervision improves language-model reasoning, but step-level correctness labels are hard to obtain for embodied agents because an action's quality depends on its downstream environmental consequences. We introduce ForkPRM, which turns that dependence into a supervision signal: during training, candidate actions are executed from the same observed navigation history and each resumes the same frozen continuation policy, so the measured utilities of the resulting branches give baseline-relative process labels without human annotation. These labels train a candidate-set verifier that scores candidates at application time and selectively overrides or defers to the policy action without executing additional environment forks. On R2R vision-and-language navigation, shared-prefix supervision raises ordinary pairwise ranking over unpaired outcome regression in the same five-seed supervision panel and label budget (+2.4 pp, positive in all five seeds; +7.6 pp over a cross-state permutation control), but these ranking gains do not extend to utility-weighted accuracy, rescue recall, or harm rate. A one-shot evaluation on scenes reserved from the verifier's splits—drawn, like the policy's own training, from the R2R training-environment pool—yields a small branch-selection gain whose confidence interval includes zero against a much larger budget-matched oracle; ranking accuracy is therefore not yet policy utility.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, process supervision, process reward models, counterfactual supervision, verifier-guided inference, embodied agents, R2R-CE, action ranking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aQT4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ForkPRM introduces executable counterfactual supervision for embodied vision-and-language navigation. By branching the environment in a simulator from shared histories under a frozen continuation policy, it generates baseline-relative process labels without human annotation. These train a verifier that ranks candidate actions. While shared-prefix supervision improves pairwise ranking, its branch-selection gains remain small compared to an oracle, showing that ranking accuracy does not yet translate to full policy utility.

Critical strengths

The proposed method assumes the availability of a reliable simulator. If the assumption holds, the verifier can be trained effectively, and the result seems to validate that. But the paper does not address the fundamental challenge of policy learning, as explained below.

Critical weaknesses

This paper presents a PRM, implemented as a verifier, trained on process labels generated by a simulator. So, the quality and fidelity of the simulator directly bound the reliability of the training labels. If the simulator's physics, rendering, or navigation graph dynamics diverge too much from reality (or even from ideal reasoning), the verifier learns from noisy or flawed counterfactual outcomes.

Therefore, while the tech seems useful in some settings with a reliable simulator, as validated in the paper, this study does not address the fundamental challenges of policy learning for physical AI -- the lack of reliable simulators in many settings. Notice that if such a reliable simulator is available for a setting, it is easy to generate unlimited amounts of training data for policy learning, and whether to use PRM is not that important because the original goal of PRM is to improve the sample efficiency of policy learning.

Questions for authors

it would be useful to evaluate the effectiveness of ForkPRM on different settings where simulators are of different quality. It would give us a sense what ForkPRM can achieve beyond the simulator.

Reviewer_bnc8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ForkPRM, a framework that uses executable counterfactual rollouts to generate process-level supervision for embodied navigation. By comparing candidate actions from the same navigation history, the method trains a verifier to selectively override the original policy without additional rollouts at inference time. While shared-prefix supervision improves ordinary pairwise ranking accuracy, the improvement does not translate into reliable gains in offline branch selection.

Critical strengths
  1. Interesting supervision approach. The paper uses executable counterfactuals to obtain process-level supervision without human annotations. The shared-prefix approach improves ordinary pairwise ranking accuracy by 2.4 percentage points over unpaired outcome regression in the controlled comparison.
  2. Careful evaluation. The authors include multiple random seeds, ablation studies, and a pre-registered confirmation experiment. The negative results are also reported transparently.
Critical weaknesses
  1. The main concern is that the improvements in ranking accuracy do not lead to meaningful action-selection gains. ForkPRM achieves only +0.10 SPL, with a 95% confidence interval of [-0.84, +1.23], and captures just 2.0% of the budget-matched oracle gain. Moreover, the evaluation is limited to offline single-override decisions, without closed-loop testing or evaluation on standard unseen environments. It therefore remains unclear whether the verifier can improve actual navigation performance.
  2. The complete ForkPRM achieves lower mean pairwise ranking accuracy than the variant without visual features (0.686 vs. 0.711), although the difference is within the reported seed variability. The other ablations also show no clear benefit from the individual components. This makes it difficult to assess how much the proposed architecture contributes beyond the shared-prefix supervision itself. Given the existing work on rollout-based action-value estimation and paired comparisons, the technical novelty also appears somewhat limited.
Questions for authors
  1. Why does the variant without visual features achieve higher mean ranking accuracy than the full ForkPRM? Is there stronger evidence that the additional architectural components improve verification performance?
  2. What explains the gap between improved pairwise ranking accuracy and the limited action-selection gains? Beyond the oracle analysis already provided, can the authors identify whether this is mainly due to representation limitations, the training objective, or intervention calibration?
Reviewer_UuDqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ForkPRM obtains process supervision by executing candidate actions from the same navigation history and continuing each branch with a frozen policy. Differences in final SPL provide baseline-relative labels for a small candidate-set verifier, which selectively overrides the policy without additional forks at application time. The experiments compare supervision strategies and evaluate single-override branch selection on a held-out confirmation set. Shared-prefix training improves ordinary pairwise ranking in the no-additional-visual-feature variant, while the full verifier yields a small offline gain of +0.10 SPL with a confidence interval that includes zero. The paper focuses on the gap between ranking accuracy and intervention utility.

Critical strengths
  1. The supervision protocol is carefully controlled. Candidate actions share a replayed history, the continuation policy is frozen, and the authors describe checks on the replay state and intervention traces. The five-seed comparison against unpaired outcome regression provides useful evidence about the effect of paired supervision.

  2. The paper reports its negative results clearly. It separates ordinary ranking from utility-weighted accuracy, rescue recall, and intervention gain, and explicitly states that most pre-registered criteria were not met. The frozen confirmation protocol and scene-level confidence intervals make the limits of the results easier to assess.

Critical weaknesses
  1. The paper identifies a ranking-utility gap but provides limited insight into what causes it. Paired training raises ordinary accuracy from 0.687 to 0.711, while utility-weighted accuracy falls from 0.674 to 0.660 and rescue recall does not improve. The hybrid-oracle analysis shows shortcomings in both deciding when to intervene and choosing the replacement action. The underlying explanations, including representation quality, label noise, objective mismatch, and calibration, remain largely unresolved. Rollout-based action values and paired comparisons are established ideas, so the contribution depends heavily on what this failure analysis teaches. The current study documents a narrow improvement and a downstream failure, but offers little evidence to identify the mechanism or a design principle that would carry beyond this setup.

  2. The scene-stability claim does not reconcile with the per-scene results. Table 8 lists 16 routes for every scene, and its twelve reported values average to approximately +0.099 SPL, matching the headline +0.10. Removing scene kEZ7cmS4wCh, whose reported value is +4.93, gives an average of approximately -0.34 over the remaining eleven scenes. These numbers imply 11/12 positive leave-one-scene-out estimates, whereas Tables 3 and 9 and the main text report 12/12. The authors should explain whether Table 8 uses a different normalization or correct the stability result. This matters because scene stability is used to distinguish the full verifier from ForkPRM-MLP despite their uncertain aggregate difference.

Questions for authors
  1. Are the ranking improvements concentrated in pairs with small utility differences? A breakdown by realized gain magnitude and rescue/harm category would help explain why ordinary accuracy improves while the utility-related metrics do not.

  2. How well do the score margin and the auxiliary delta head predict realized intervention gain on the development data? Does using the predicted delta instead of the score margin improve selection at the same intervention budget? This could help distinguish a ranking problem from a calibration or gating problem.

  3. The labels depend on the continuation policy. When transferring to the second checkpoint, could the failure partly reflect a change in the true ordering of action values? Comparing the two continuation policies from a common prefix and candidate set would clarify this before attributing the result to learning the source policy's habits.

  4. Please reconcile the 12/12 leave-one-scene-out result with Table 8 and provide the corresponding recomputed values. How are the per-scene entries normalized relative to the pooled estimate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.