- Rating
- 2(+1)
- Confidence
- 3(+1)
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduce RoboStep, a hierarchical reward system consists of 2 VLMs, a GPT 5.5 frontier model split the task into stages and generate dense reward protocol inside the stage, a Qwen opensource VLM check each frame if the next stage is achieved. The experiments show that RoboStep helps manipulation policy training with RL.
Critical strengths
The method is a training-free reward system, utilizing frontier VLM capabilities.
Critical weaknesses
-
The method of split the long horizon task into stages/subtask is not new, SARM [1] has showed that this design is helpful for reward modeling and down-stream policy learning. RoboStep seems build on the same principle but not compare with SARM.
-
Though authors have designed to use different tiers of model, but inference a 9B VLM each frame (or every several frames) still seem expensive and unscalable. Also there is no comparison with the relatively smaller reward models on reward qualities.
-
Relying on VLM solely for stage splitting and dense reward function generation is unreliable. Also many details are not clear for the examples in A-3, A-4, how does Frame-VLM generate such detail rules by only read the task instruction without viewing a task demo ep?
-
The authors only use GPT & Qwen combination for 2 VLMs, as an ablation study should at least use 2 frontier models and 2 opensourced models, together as 4 combinations, showing that the ability comes from RoboStep system design rather than purely the model ability.
-
The real experiment is from real-to-sim-to-real, not directly showing that the reward system can work on real robot eps.
-
The Figure 1 and 2 are very low quality, the color, font size, and position make the text very unclear.
Reference: [1] Chen, Qianzhong, et al. "Sarm: Stage-aware reward modeling for long horizon robot manipulation." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
How you make sure the Frame-VLM generation quality? Have you had any failure case that it’s bad generation cause the task harder to learn from sparse reward?
7 comments
Response to Reviewer wJ1p ( Weakness 2/3 & Question )
Weakness 2
Though authors have designed to use different tiers of model, but inference a 9B VLM each frame (or every several frames) still seem expensive and unscalable. Also there is no comparison with the relatively smaller reward models on reward qualities.
Thank you for raising these important concerns about inference cost and reward quality!
-
Inference Cost Clarification.
We would like to note that Gate-VLM is not invoked at every frame. Frame-VLM generates reward functions offline, while Gate-VLM is queried only when transition candidates are triggered. Between queries, the persistent step state is maintained and dense rewards are computed without VLM inference (Appendix A-2, C-3).
-
Model Scale and Reward Quality.
We fully understand the reviewer's concern about model scale and reward quality. For additional context, we note that several related visual-reward methods also use VLMs of comparable sizes:
| Method | SOLE-R1 | RoboMeter | RoboReward | TOPReward | LRM | RoboStep |
|---|---|---|---|---|---|---|
| Model Size | 8B | 4B | 4B / 8B | 8B | 8B | 9B |
We further discuss expanded smaller-model experiments along with our response to Weakness 4.
- Reward Quality. We suggest that reward quality in our framework may be related to step-completion verification accuracy and downstream policy performance. We hope that our false-acceptance/false-rejection rate (FAR/FRR) analysis (Sec. 5.3) and PPO success rates across 75 manipulation tasks (Table 2) could offer some preliminary evidence in this regard.
Weakness 3 & Question
Relying on VLM solely for stage splitting and dense reward function generation is unreliable. Also many details are not clear for the examples in A-3, A-4, how does Frame-VLM generate such detail rules by only read the task instruction without viewing a task demo ep?
How you make sure the Frame-VLM generation quality? Have you had any failure case that it’s bad generation cause the task harder to learn from sparse reward?
Thank you for these thoughtful questions about reward generation and its reliability!
-
Frame-VLM Input Clarification.
We would like to clarify that Frame-VLM does not rely solely on the task instruction. As illustrated in Fig. 2(a), its initial inputs include the task description, available simulator state/API interfaces, and reward design constraints.
Appendix A-3 provides the prompts used for reward generation and visual verification, while Appendix A-4 presents representative generated reward programs. We further provide per-task Frame-VLM inputs, generated step decompositions, and dense reward components for all 75 tasks in Appendix G-2 (Table 24).
-
Related Work. Previous works in Programmatic and Structured Reward Design have demonstrated the potential of language models for reward generation and task decomposition, such as Text2Reward (ICLR 2024), Eureka (ICLR 2024), Code-as-Reward (ICML 2024) and PRM4RL (NeurIPS 2025).
-
Frame-VLM Generation Quality. We fully agree that generated reward quality is important. We also recognize the increasingly strong capabilities of recent foundation models in generating executable reward code.
- In our initial Meta-World reward compilation using GPT-5.5 (xhigh), 19/50 tasks already met the 50% validation-success threshold without further reward revision.
- As detailed in Appendix A-1, we adopt validation-guided reward refinement, similar to iterative reward improvement in Eureka and Text2Reward. When PPO validation success is below 50%, Frame-VLM revises the previous stage/reward design using training diagnostics including success rate and failure modes. We allow up to 5 trials to match Eureka's default iteration count, and use the final accepted or 5th-round reward program for evaluation. The distribution of compilation rounds across our 75 tasks is:
| Benchmark | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 | Total |
|---|---|---|---|---|---|---|
| MetaWorld | 19 | 11 | 8 | 7 | 5 | 50 |
| ManiSkill | 6 | 2 | 3 | 2 | 6 | 19 |
| SoftGym | 1 | 1 | 0 | 1 | 3 | 6 |
We sincerely hope these clarifications and additional statistics help address the reviewer's concerns about the inputs, generation process and reliability of Frame-VLM.
Response to Reviewer wJ1p ( Weakness 4/5/6 )
Weakness 4
The authors only use GPT & Qwen combination for 2 VLMs, as an ablation study should at least use 2 frontier models and 2 opensourced models, together as 4 combinations, showing that the ability comes from RoboStep system design rather than purely the model ability.
Thank you for this valuable suggestion on cross-model generalizability!
-
System Design Clarification.
We view VLM capability and system design as complementary rather than mutually exclusive. VLMs provide semantic understanding, while RoboStep organizes it into persistent step state, explicit reward routing, and step-conditioned policy learning.
To examine the contribution of system design under fixed model capabilities, Table 1 uses the same Frame-VLM-generated reward components and independently varies stepwise/all-dense reward structure and step-index policy inputs. Table 2 further includes Gate-Rule as a reference to distinguish visual verification from the underlying stepwise framework.
-
Cross-Model Experiments.
We fully agree that cross-model evaluation is also a valuable way to examine robustness. We used a 10-task MetaWorld subset to additionally evaluate different Frame-VLM and Gate-VLM combinations. The mean downstream PPO success rates are:
| Frame-VLM / Gate-VLM | Qwen3.5-9B (Main) | Qwen3.5-27B | Qwen3-VL-8B | Qwen3-VL-4B | Qwen3-VL-32B |
|---|---|---|---|---|---|
| GPT-5.5 xhigh (Main) | 38.31% | 46.66% | 39.25% | 30.22% | 34.51% |
| GPT-6 Astra xhigh | 40.07% | 45.18% | 41.37% | 35.67% | 36.12% |
| Claude Fable 5.1 | 41.13% | 47.04% | 40.21% | 37.45% | 35.08% |
These results suggest that RoboStep can support policy learning across different Frame-VLM and Gate-VLM combinations. We hope it provides some additional evidence that RoboStep is not restricted to the original GPT-5.5 + Qwen3.5-9B combination.
We sincerely thank the reviewer again for advice on this additional evaluation.
Weakness 5
The real experiment is from real-to-sim-to-real, not directly showing that the reward system can work on real robot eps.
Thank you for raising this distinction.
- Real-World Deployment Clarification.
RoboStep is designed for from-scratch PPO training in simulation, followed by real-world deployment without demonstrations or real-world policy updates. Recent advances in real-to-sim-to-real scene reconstruction make this a promising option. Importantly, Gate-VLM still operates on real RGB observations during deployment, maintaining the persistent step state through visual verification (Sec. 4.3, Appendix D).
- Different Real-World Learning Settings.
We carefully reviewed related visual-reward methods that explore different real-world policy-learning protocols:
| Method | Real-World Policy Learning Setting |
|---|---|
| SARM | Demonstration-supervised stage-aware reward modeling & reward-aligned BC fine-tuning of pretrained policy |
| RoboReward | BridgeData V2-pretrained diffusion policy, 20 warm-start rollouts, followed by online DSRL-SAC |
| VLAC | Approximately 100 teleoperated trajectories for policy initialization, followed by real-world PPO with additional human-guidance |
| ReWiND | Demonstration-based reward and policy pretraining, followed by real-world online RL with human-provided success signals |
| TOPReward | Policy pretrained on 200 hours of data, followed by reward-weighted BC using 50 demonstrations per task |
We greatly appreciate these complementary approaches, each with different supervision and real-world interaction requirements, while our focus is on exploring whether structured visual rewards can enable demonstration-free simulation training and direct real-world transfer.
Weakness 6
The Figure 1 and 2 are very low quality, the color, font size, and position make the text very unclear.
Thank you for your feedback on the readability of Figures 1 and 2.
As RoboStep involves several interconnected components and experimental factors, we invested considerable effort in illustrating their interfaces and interactions in Figures 1 and 2. We regret that the figures may not have appeared as clear as intended.
Following your suggestions, we have revised the font sizes and color contrast in the updated PDF. We sincerely hope these improvements help address your concerns.
Summary
We sincerely appreciate your time and thoughtful feedback throughout this review and would be happy to clarify any remaining questions. If our responses have helped address your concerns, we would be very grateful if you might consider updating your score and confidence!
Response to Reviewer wJ1p ( Weakness 1 )
Thanks for these thoughtful suggestions! Our responses are as follows:
Weakness 1
We would like to clarify that we do not intend to claim novelty in decomposing long-horizon tasks into subtasks or novelty in demonstrating the benefits of stage information for reward modeling and downstream policy learning.
RoboStep focuses on online policy learning without pretrained policy initialization, task-specific demonstrations or real-world RL. Therefore, we selected five visual-reward baselines from related work (SOLE-R1, LRM, RoboMeter, RoboReward, and TOPReward) whose reward models are described in their respective papers as applicable without task-specific demonstration adaptation, and evaluated them under a shared online PPO training protocol.
We greatly appreciate SARM's demonstration-supervised approach to challenging deformable manipulation tasks such as T-shirt folding, where its reward model is trained on 200 densely annotated and 500 sparsely annotated trajectories. We are concerned that directly evaluating SARM under our demonstration-free setting may not adequately reflect its intended strengths.
We see encoding task structure into reward/policy learning and learning stage/progress representations from trajectories as potentially complementary directions: