- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work uses process reward models to obtain task-progress curves and derive a set of evaluation metrics. It also developed RoboPulse to compare different progress discriminators.
Critical strengths
The approach is novel; the quality of task execution is important and can be used for downstream applications such as data filtering.
The interface is simple. It requires only the task text and an execution video.
The RoboPulse dataset is highly valuable and is expected to advance the community’s research on this problem.
The experimental setup is comprehensive, spanning different benchmarks, multiple models, and tasks.
Critical weaknesses
The default judge contains errors, particularly in detecting progress regression, where its Falling F1 is 0.61. This has a significant impact on the evaluation results. While this is not unacceptable, it seems to require an appropriate explanation.
The VLM included in the experiment omits some of the latest frontier models, such as GPT-6 series.
Questions for authors
When the judge itself has substantial errors, what exactly is the evaluation measuring? How should these errors be measured and addressed? Could downstream use of the scores introduce reward-hacking risks? These are common issues across many evaluation methods, and I believe these questions warrant some discussion.