acceptodds
Withdrawn from ICLR 2027

ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon robotic manipulation requires dense feedback that reflects how a task advances through its procedural stages, not merely whether the final outcome is successful. Existing reward models often rely on trajectory-level success labels or time-based interpolation, which can conflate elapsed time with true task progress and therefore fail to capture unfinished steps, stagnation, and failure states. We present ProcVLM, a progress-aware vision-language model that learns procedure-grounded progress as a dense reward signal for manipulation. Rather than deriving progress from terminal outcomes or temporal proxies, ProcVLM grounds progress estimation in procedural structure and intra-stage visual change, and further adopts a reasoning-before-estimation paradigm that infers the remaining atomic actions before estimating task progress. Specifically, we construct this supervision by synthesizing frame-level subtask-semantic annotations, assigning progress budgets according to subtask structure, and distributing each budget based on intra-subtask visual change. To train ProcVLM at scale, we build a standardized procedural supervision synthesis pipeline and construct ProcCorpus-60M from 30 embodied datasets with 60M annotated frames, from which we derive ProcVQA for procedure-aware pretraining, with progress estimation as the central task alongside action segmentation and future planning. Experiments on ProcVQA and reward-model benchmarks show that ProcVLM improves embodied procedural reasoning and yields more discriminative trajectory-internal progress estimates than representative baselines, supporting its use as a dense reward model for downstream reward-guided policy optimization.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reward modeling, robot learning, robotic manipulation, reinforcement learning, embodied vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Lii5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ProcVLM, a VL model for estimating task progress in robotic manipulation. The approach first identifies the current execution stage and remaining actions, then predicts a continuous progress score. ProcVLM is trained jointly on action segmentation, future planning, and progress estimation, with a dedicated value head for continuous prediction. Experiments evaluate procedural understanding, reward estimation, adaptation from a single demonstration, and downstream policy optimization in simulation and on a real robot.

Critical strengths
  1. The paper addresses an important limitation of outcome-only and time-based supervision: neither reliably captures intermediate progress when robots stall, retry actions, or recover from failures.
  2. The annotation pipeline scales across diverse robotic datasets. Its hierarchical processing and reuse of keyframe reasoning make large-scale procedural supervision more practical.
Critical weaknesses
  1. The paper does not report the number of evaluation seeds or standard deviations, making it difficult to assess the consistency of the results.
  2. The downstream evaluation covers too few environments and tasks to establish the method’s general effectiveness. The lack of information about independent runs further limits confidence in the reported improvements.
  3. The contribution of the progress-target construction is insufficiently isolated. The reported ablations examine pretraining and reasoning supervision but do not directly compare the proposed targets against time interpolation, equal-weight subtask progress, or alternatives without visual-change weighting. These comparisons are important because target construction is a central methodological contribution.
Questions for authors
  1. How many independent training seeds and evaluation episodes were used for each policy-learning experiment? Please report standard deviations or confidence intervals to clarify whether the improvements are consistent across runs.
  2. Do the gains generalize beyond the two simulation tasks and the real-world block-stacking task? Can the authors provide additional evaluations across tasks requiring different manipulation skills, longer planning horizons, and recovery from failures?
  3. How does the proposed progress-target construction compare with time interpolation, equal-weight subtask progress, and a variant without visual-change weighting when using the same training data and model?
2 comments

W1 & Q1: We thank the reviewer for raising this concern. Each policy-learning experiment was conducted with a fixed training seed and a fixed evaluation-seed configuration. Given the substantial computational and time costs of repeating policy training and evaluation across multiple independent seeds, we did not conduct such repetitions and therefore do not report standard deviations across training runs.

Nevertheless, the reported results show substantial improvements over the baselines under the evaluated configurations, as illustrated in Tables 1–3 and Figure 3. For instance, Figure 4(b) shows a pronounced advantage in convergence. Following the standard RoboTwin evaluation protocol, each evaluation comprises 100 episodes with different environment seeds, so the reported performance aggregates results across multiple initial conditions. This reduces dependence on any single test initialization, and we therefore believe that the current results provide meaningful evidence of the method’s effectiveness. We do not share the reviewer’s concern regarding run-to-run consistency.

W3 & Q3: We must emphasize that progress-label construction is only one component of our method. As discussed in the Introduction, our central premise that “time is not progress” informs several design choices, including reasoning before estimation, large-scale multi-task pretraining, and procedure-grounded label construction.

We do not position a particular labeling scheme as the primary contribution of this work. Instead, our main contributions concern the cross-task transfer enabled by reasoning and large-scale pretraining, together with the ability to generalize reward estimation from successful demonstrations to failed or stalled executions. Our ablation studies and the controlled comparisons in Section 4.2 provide evidence supporting these conclusions.

We therefore believe that focusing narrowly on the labeling scheme does not adequately reflect the scope of our methodological contributions, and we respectfully disagree with this criticism.

W2 & Q2: We acknowledge this limitation. We are currently conducting RL experiments on a multi-step real-robot task, including comparisons with other baseline reward models. We will report the results as soon as these experiments are completed.

Thank you for the response, but the evaluation remains insufficiently thorough, and the additional experimental results have not yet been provided. I therefore maintain my original rating.

Reviewer_eWPoOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ProcVLM, a 2B vision-language model for procedure-grounded progress estimation in robotic manipulation. It combines reasoning about remaining actions with textual progress prediction and a continuous regression head. The authors construct ProcCorpus-60M and ProcVQA through automated procedural annotation, deriving progress targets from subtask budgets and within-subtask visual change. Evaluations cover procedural understanding, zero-shot and few-shot reward modeling, and downstream policy learning on two RoboTwin tasks and one real-robot task. Results support the utility of procedure-aware pretraining and task-specific adaptation.

Critical strengths
  1. The method connects progress estimation with explicit reasoning about current and remaining subtasks. Its procedure-defined targets offer a structured alternative to global time interpolation, incorporating both stage boundaries and within-stage visual change.

  2. The annotation pipeline produces large-scale procedural supervision across 30 datasets. The combination of automated annotation, a documented quality audit, and curated refinement provides a practical approach to training a compact procedural VLM.

  3. Evaluation extends beyond progress prediction to few-shot adaptation and downstream policy learning. Ablations examine procedure-aware pretraining and reasoning supervision, while simulation RL and real-robot fine-tuning provide evidence that the learned progress signals can support policy improvement.

Critical weaknesses
  1. Within-subtask progress is derived from perceptual change, which need not correspond to task advancement during retries, irrelevant motion, or viewpoint changes. The experiments do not isolate the contribution of this construction through comparisons with simpler subtask-based targets, leaving its reliability and additional value unclear.

  2. The zero-shot comparison uses a blank contrastive anchor for RoboDopamine rather than its intended comparison inputs. Although this limitation is acknowledged, the resulting scores do not cleanly distinguish reward-model capability from sensitivity to the modified input format.

  3. The annotation audit reveals substantial variation in supervision quality across datasets. Only 49.44% of audited trajectories satisfy the strict all-subtasks-and-boundaries criterion, although this is not a frame-level accuracy measure. Curated refinement mitigates this concern, but the effect of annotation noise on progress estimation and downstream rewards is not directly quantified.

Questions for authors
  1. How are incomplete executions, failed attempts, and task regression represented in the synthesized progress targets? Equation (1) accumulates nonnegative visual changes and normalizes by the trajectory total. Is it applied only to successful demonstrations, and how are unfinished task stages prevented from being assigned full completion?

  2. Can you evaluate RoboDopamine with valid comparison frames following its intended input protocol on the same evaluation trajectories? This would help determine how much of the reported zero-shot gap is attributable to the blank-anchor construction.

2 comments

W1 & Q1: We thank the reviewer for raising these concerns. Our procedural annotation and pretraining use only successful expert demonstrations from the datasets listed in Table 6. These demonstrations are not expected to contain substantial ineffective retries or failure recovery, and we adopt monotonic progress as a supervision assumption. Equation (1) is applied only to these successful demonstrations; failed or incomplete executions are excluded from target construction and are therefore not normalized to full completion.

This assumption concerns the construction of training targets, while our experiments evaluate generalization beyond successful demonstrations. Specifically, Section 4.2.2 evaluates ProcVLM on RoboFAC, which includes both successful and failed executions. Adaptation using only one successful demonstration per task yields a 47.5% average relative improvement across the three primary metrics over zero-shot ProcVLM. On failed trajectories, its MAE is 45.3% lower than that of the pretrained RoboDopamine baseline and comparable to Robometer’s under the same one-shot adaptation setting. These results support the ability of pretraining with reasoning annotations on successful demonstrations to facilitate generalization to failed executions.

We further provide three qualitative examples showing predicted progress curves for trajectories involving failures or retries: 1, 2, and 3. Together with the quantitative results, these examples provide evidence of generalization beyond successful training trajectories. We will clarify the scope of our training supervision and distinguish its assumptions from the model’s demonstrated generalization capability in the revised manuscript.

W2 & Q2: We would like to clarify that, under the ProcVQA preprocessing pipeline, each test sample contains only a local window of eight consecutive frames. Consequently, the first frame of the full trajectory is unavailable as a reference input to RoboDopamine in this experiment. We therefore use its supported blank-anchor mode.

To address this limitation, we conduct additional comparisons against RoboDopamine on the RoboTwin test set, with RoboDopamine evaluated using its native input protocol. On the move_can_pot task, GRPO training with ProcVLM exhibits substantially faster convergence than with RoboDopamine. We believe these additional results adequately address the concerns regarding the blank-anchor evaluation setting.

W3: We thank the reviewer for raising the concern about annotation quality. Table 11 in Appendix G reports trajectory-level acceptance rates across different data sources under human review. The overall acceptance rate of 49.44% reflects a stringent criterion requiring all subtasks and their boundaries to be correct: a trajectory is rejected if any subtask boundary deviates from the human-annotated boundary by more than one second. This is a whole-trajectory acceptance rate, not a frame-level accuracy measure.

To mitigate annotation noise, the second stage of pretraining uses only human-screened trajectories that satisfy this criterion. Figure 6 shows that, on the in-distribution test set (test_full), the progress prediction metrics VOC and KT, together with the task-completion classification metric MCC@95, improve rapidly and substantially during this stage. These improvements support the value of refinement with higher-quality supervision and are consistent with its role in mitigating annotation noise, although they do not independently quantify the effect of noise. We will clarify the audit criterion and the role of human-screened refinement in the revised manuscript.

Thank you to the authors for their detailed responses. I appreciate their efforts, but some of my main concerns remain:

W1: Thanks for clarifying that equation (1) is only used for successful demonstrations, which resolved my doubts about failed or incomplete trajectories being normalized to full completion. The RoboFAC results also support that the model can generalize to failed executions. However, these results do not yet show that assigning intra-stage progress by visual variation is more efficient than simple intra-stage linear interpolation. Are there comparisons that keep the same subtask boundaries and only replace the way progress is calculated within a phase? W2: I agree that the RoboTwin experiments under the native input protocol provide useful complementary evidence. However, what is evaluated here is the RL performance of the reward model after task adaptation, which cannot directly illustrate the influence of the blank reference map on the original zero-shot progress prediction results. If it is not possible to supplement the effective reference input on the original test trajectory, it is recommended to explicitly limit the conclusion of this comparison to the current local window setting to avoid directly interpreting the score difference as a model ability gap.

Therefore, I will keep my rating

Reviewer_cT2vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ProcVLM, a vision-language model that learns procedure-grounded progress as a dense reward signal for robotic manipulation. Instead of deriving progress only from linear time-based labels, ProcVLM grounds progress estimation in procedural structure and intra-stage visual change, and further adopts a reasoning-before-estimation paradigm. The authors construct ProcCorpus-60M from 30 embodied datasets with approximately 60M annotated frames and convert the annotations into ProcVQA. Experiments on ProcVQA and reward-model benchmarks show that ProcVLM improves embodied procedural reasoning and yields more discriminative trajectory-internal progress estimates.

Critical strengths
  1. The motivation is clear and relevant: dense feedback from a reward model should reflect how a task advances through its procedural stages, rather than rely only on trajectory-level success labels or time-based interpolation.

  2. The construction of ProcVQA dataset is solid. And the pipeline combines video-level subtask annotation with frame-level reasoning across many real-robot and simulation datasets.

Critical weaknesses
  1. The definition of the progress labels needs further clarification. Equation (1) defines progress as the normalized accumulation of non-negative local visual changes, weighted by subtask duration. These labels are non-decreasing and cannot directly represent a loss of task progress, such as when an object is dropped before placement or moved away by a person. This also raises questions about how ineffective retries and failure recovery are labeled.

  2. The experimental evidence is insufficient. (1) The main progress metrics reported do not fully test the claim that the model learns task progress rather than timeline. As VOC measures temporal ordering within trajectories, while EPR measures the resolution of predicted scores. Neither directly measures whether the progress values are correct. (2) The zero-shot comparison with RoboDopamine and Robometer is difficult to interpret. RoboDopamine is evaluated with a blank image as its before-state anchor, which differs from its intended input format and may limit its performance. Even under this setting, ProcVLM has only a small VOC advantage (0.7282 vs. 0.7156; Table 2), leaving it unclear whether this advantage would hold if the evaluation settings are more complete. In addition, Robometer achieves a higher EPR than ProcVLM: 6.7212 v.s. 6.1152. (3) The ablations do not clearly separate the benefit of the proposed progress labels from the benefit of additional training data.

Questions for authors
  1. Could the authors provide more detailed statistics on subtask and temporal boundary errors, and analyze how these errors affect progress estimation?

  2. The authors should evaluate more real-robot rollouts and longer tasks, and compare ProcVLM with other reward models under the same real-robot fine-tuning setup. These experiments would provide stronger evidence of its effectiveness as a reward model.

  3. See the concerns raised in Critical Weaknesses.

2 comments

W1: We thank the reviewer for this comment. Our procedural annotation and pretraining use only successful expert demonstrations from the datasets listed in Table 6. These demonstrations are not expected to contain substantial ineffective retries or failure recovery, so we adopt monotonic progress as a supervision assumption.

Crucially, our generalization experiments show that pretraining on successful demonstrations with reasoning annotations supports progress estimation on trajectories involving failures, retries, and other nonlinear progress patterns. Section 4.2.2 provides evidence for this capability on RoboFAC, a benchmark designed to evaluate failure correction that includes both successful and failed demonstrations. After adaptation using only one successful demonstration per task, ProcVLM achieves an average improvement of 47.5% across the three primary metrics over its zero-shot performance. On failed trajectories, ProcVLM achieves a 45.3% relative improvement in MAE over the pretrained RoboDopamine baseline, while achieving performance comparable to Robometer under the same one-shot adaptation setting.

We would like to further present three cases showing progress curves for trajectories involving failures or retries: 1, 2, 3. These results can address the concern about generalization beyond successful training trajectories.

W2:

(1) We'd like to emphasize that VOC is an established primary metric for evaluating progress prediction, introduced by GVL and subsequently adopted by OpenGVL, TOPReward, Robometer and Robo-Dopamine. It evaluates the correctness of relative progress ordering on expert trajectories without requiring different models to share an identical numerical progress parameterization. EPR measures the resolution of predicted progress scores; we report it only on ProcVQA as an auxiliary assessment of the granularity of model outputs.

(2) Robometer follows its standard deployment and inference protocol throughout all evaluations. The processed ProcVQA data contain only images within the current temporal window, without access to the initial frame of the full trajectory. Using the first frame of the current window as the initial-state anchor would substantially degrade RoboDopamine’s performance, since it estimates relative progress. We therefore use a blank image anchor in this setting, which is also an officially supported usage of RoboDopamine. To address this limitation, we further compare ProcVLM with RoboDopamine on RoboTwin in Section 4.3, where RoboDopamine is evaluated using its native protocol.

(3) We must clarify that the primary contribution of this work is not a particular method for constructing progress labels. Rather, it is to demonstrate that incorporating procedure-based priors into progress reward learning—including the design of reasoning steps, progress labels, and multi-task learning—improves generalization and the ability to handle rollouts with nonlinear progress. We validate this finding through large-scale pretraining and evaluation across multiple settings.

Q1: Table 11 in Appendix G reports the accuracy of subtask boundary annotations generated by the initial annotation model across different data sources, measured by the trajectory-level acceptance rate under human review. A trajectory is rejected if any subtask boundary deviates from the boundary identified by human annotators by more than one second. Under this stringent criterion, the acceptance rate across the full pretraining dataset is approximately 50%, indicating that roughly half of the pretraining trajectories have accurate boundary annotations for every subtask.

Importantly, in the second stage of pretraining, we further refine the model using only data that pass human screening, whose annotation accuracy should, in principle, approach 100% under this criterion. Figure 6 presents the model’s evaluation results throughout the two-stage pretraining process. On the in-distribution test set (test_full), both progress prediction metrics, VOC and KT, and the task-completion classification metric, MCC@95, improve rapidly and substantially during the second stage. These results give us confidence that human review and subsequent refinement effectively mitigate the impact of boundary annotation errors.

Q2: We thank the reviewer for this suggestion. We are currently conducting an RL training experiment on a multi-step real-robot task, including comparisons with other baseline reward models. We will report the results as soon as the experiments are completed.

Reviewer_cT2vOpenReview ↗

Comment by Reviewer

I thank the authors for their response and efforts to address my concerns. While some of my concerns have been resolved, several concerns regarding the experimental evaluation remain unresolved. I will therefore maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.