acceptodds
Withdrawn from ICLR 2027

PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robotic policies are commonly evaluated by binary success rates, which compress an entire execution into a terminal outcome and provide limited information about the execution process. We present PRM-as-a-Judge, a process-level evaluation paradigm that uses Process Reward Models (PRMs) to estimate dense task progress from robot rollout observations and represent each execution as a progress curve. Based on this curve, we introduce the OPD (Outcome–Process–Diagnosis) metric system, which organizes trajectory evaluation around three sequential questions: (1) how far the execution reaches (Outcome), (2) how efficiently it advances (Process), and (3) how progress deteriorates through regression or stagnation (Diagnosis). To evaluate progress judges before policy auditing, we introduce RoboPulse, an interval-level benchmark that measures their ability to identify whether task progress increases or decreases over temporal intervals. Experiments across diverse tasks and data sources show a clear advantage of specialized PRM judges over general-purpose vision-language model (VLM) judges in task-progress assessment. Finally, we apply PRM-as-a-Judge to large-scale policy auditing in both simulation and real-world rollouts, revealing: (1) stage-wise bottlenecks, (2) differences in successful execution quality, and (3) distinct regression and stagnation patterns that are not captured by terminal success rates. We will release RoboPulse and the complete PRM-as-a-Judge evaluation toolkit.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic evaluation, process reward models, robot manipulation, progress estimation, embodied AI, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RcA8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work uses process reward models to obtain task-progress curves and derive a set of evaluation metrics. It also developed RoboPulse to compare different progress discriminators.

Critical strengths

The approach is novel; the quality of task execution is important and can be used for downstream applications such as data filtering.

The interface is simple. It requires only the task text and an execution video.

The RoboPulse dataset is highly valuable and is expected to advance the community’s research on this problem.

The experimental setup is comprehensive, spanning different benchmarks, multiple models, and tasks.

Critical weaknesses

The default judge contains errors, particularly in detecting progress regression, where its Falling F1 is 0.61. This has a significant impact on the evaluation results. While this is not unacceptable, it seems to require an appropriate explanation.

The VLM included in the experiment omits some of the latest frontier models, such as GPT-6 series.

Questions for authors

When the judge itself has substantial errors, what exactly is the evaluation measuring? How should these errors be measured and addressed? Could downstream use of the scores introduce reward-hacking risks? These are common issues across many evaluation methods, and I believe these questions warrant some discussion.

Reviewer_LmzSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses a prior reward progress model (RoboDopamine) to predict the progress curve for the recorded rollout videos offline. The predicted curve is summarized into five different statistics Outcome (MC, MP), Process (PPL), Diagnosis (CRA, STR). Beyond metrics, it also introduces RoboPulse, a benchmark to evaluate specialized reward progress model vs general purpose VLMs on intervals with task progress increase and decrease. It applies the pipeline to RoboDojo benchmark with different rollouts from different policies.

Critical strengths
  1. RoboPulse's benchmark covers both simulation and real world rollouts from the common benchmark like RoboDojo and Robotwin and covers a set of different policies and ablations.
  2. The OPD metrics is straightforward and is applicable to any progress curve so the framework can get better as better progress reward models get improved.
Critical weaknesses
  1. The proposed and evaluated downstream application is narrow in this paper on how the progress reward model can be used. Process reward models have been used in many downstream applications in robotics, e.g., test-time verification for action selection (ROVER, RoboMonkey) and data curation (RoboReward, OpenGVL) and RL post-training (VLAC, RoboDopamine). In this paper, the progress reward model is only used for post-hoc summary and only used in the offline settings to evaluate the recorded rollouts. Therefore, the limited scope may weaken the contribution of the work to the community.

  2. The selection of general purpose VLM baselines seem not to be the state-of-the-art models. For example, GPT-6 Astra has demonstrated the strong capabilities of spatial understanding and Claude Fable 5 has also been much stronger than the previous 4.6 version. The conclusion drawn for general purpose VLM models are not as good as specialized models may not hold true given the recent models. Therefore, it would make the conclusion stronger if the experiments can be done with more recent models.

Questions for authors
  1. There seems to be limited qualitative examples to demonstrate the rollouts and different models as well as the proposed metrics in appendix. If the paper can include more qualitative examples to visualize the success and failures of different progress reward models that would be more informative to understand those models beyond the metrics.
  2. It is unclear to me why those metrics are necessary and sufficient for the summary analysis for those different models. The paper claims the decomposition of the outcome-process-diagnosis is necessary but doesn't motivate enough why those should be the decomposition.
Reviewer_8PKjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRM-as-a-Judge, a framework where VLMs or dedicated progress judge estimate task progress based on task instruction and rollout observations. Based on this progress estimate, the authors propose to evaluate the policy rollout with the Outcome-Process-Diagnosis (OPD) metrics. To evaluate the PRMs before using them, the authors introduce a new simulated benchmark called RoboPulse. Finally, they apply PRM-as-a-Judge to audit policies to reveal that their metrics reveal differences between the rollouts that binary success metric does not.

Critical strengths
  1. Evaluation is a hard problem, and binary success metric is not enough. The paper studies an important question of how to better evaluate and differentiate policy rollouts.
Critical weaknesses
  1. Limited significance: it is unsurprising that a binary success metric hides very important information about the differences in policy rollouts, and that more nuanced metrics are important. The authors proposed the OPD metrics, yet did not show why this is the right set of metrics to use, and why it’s better than previously proposed metrics (i.e. why can’t baseline metrics also distinguish between policy rollouts).
  2. It is unclear why RoboPulse is the right benchmark to evaluate PRMs: PRMs produce continuous progress scores, yet RoboPulse has binary labels.
Questions for authors
  1. Why is RoboPulse (and its existing annotations) the right metric and benchmark to evaluate PRMs?
  2. Does PRM-as-a-Judge help you to rank policies when they cannot be distinguished by binary success metrics? If so, how does the ranking work?
  3. Have you compared against previous baseline metrics that is more nuanced than binary success? If so, how do different metrics change the ranking of the policies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.