- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ProcVLM, a VL model for estimating task progress in robotic manipulation. The approach first identifies the current execution stage and remaining actions, then predicts a continuous progress score. ProcVLM is trained jointly on action segmentation, future planning, and progress estimation, with a dedicated value head for continuous prediction. Experiments evaluate procedural understanding, reward estimation, adaptation from a single demonstration, and downstream policy optimization in simulation and on a real robot.
Critical strengths
- The paper addresses an important limitation of outcome-only and time-based supervision: neither reliably captures intermediate progress when robots stall, retry actions, or recover from failures.
- The annotation pipeline scales across diverse robotic datasets. Its hierarchical processing and reuse of keyframe reasoning make large-scale procedural supervision more practical.
Critical weaknesses
- The paper does not report the number of evaluation seeds or standard deviations, making it difficult to assess the consistency of the results.
- The downstream evaluation covers too few environments and tasks to establish the method’s general effectiveness. The lack of information about independent runs further limits confidence in the reported improvements.
- The contribution of the progress-target construction is insufficiently isolated. The reported ablations examine pretraining and reasoning supervision but do not directly compare the proposed targets against time interpolation, equal-weight subtask progress, or alternatives without visual-change weighting. These comparisons are important because target construction is a central methodological contribution.
Questions for authors
- How many independent training seeds and evaluation episodes were used for each policy-learning experiment? Please report standard deviations or confidence intervals to clarify whether the improvements are consistent across runs.
- Do the gains generalize beyond the two simulation tasks and the real-world block-stacking task? Can the authors provide additional evaluations across tasks requiring different manipulation skills, longer planning horizons, and recovery from failures?
- How does the proposed progress-target construction compare with time interpolation, equal-weight subtask progress, and a variant without visual-change weighting when using the same training data and model?
2 comments
Thank you for the response, but the evaluation remains insufficiently thorough, and the additional experimental results have not yet been provided. I therefore maintain my original rating.
W1 & Q1: We thank the reviewer for raising this concern. Each policy-learning experiment was conducted with a fixed training seed and a fixed evaluation-seed configuration. Given the substantial computational and time costs of repeating policy training and evaluation across multiple independent seeds, we did not conduct such repetitions and therefore do not report standard deviations across training runs.
Nevertheless, the reported results show substantial improvements over the baselines under the evaluated configurations, as illustrated in Tables 1–3 and Figure 3. For instance, Figure 4(b) shows a pronounced advantage in convergence. Following the standard RoboTwin evaluation protocol, each evaluation comprises 100 episodes with different environment seeds, so the reported performance aggregates results across multiple initial conditions. This reduces dependence on any single test initialization, and we therefore believe that the current results provide meaningful evidence of the method’s effectiveness. We do not share the reviewer’s concern regarding run-to-run consistency.
W3 & Q3: We must emphasize that progress-label construction is only one component of our method. As discussed in the Introduction, our central premise that “time is not progress” informs several design choices, including reasoning before estimation, large-scale multi-task pretraining, and procedure-grounded label construction.
We do not position a particular labeling scheme as the primary contribution of this work. Instead, our main contributions concern the cross-task transfer enabled by reasoning and large-scale pretraining, together with the ability to generalize reward estimation from successful demonstrations to failed or stalled executions. Our ablation studies and the controlled comparisons in Section 4.2 provide evidence supporting these conclusions.
We therefore believe that focusing narrowly on the labeling scheme does not adequately reflect the scope of our methodological contributions, and we respectfully disagree with this criticism.
W2 & Q2: We acknowledge this limitation. We are currently conducting RL experiments on a multi-step real-robot task, including comparisons with other baseline reward models. We will report the results as soon as these experiments are completed.