acceptodds
Under review as a conference paper at ICLR 2027

RoboStep: Robot Rewards Benefit from Tracking Task Progress Step by Step

Abstract

Progress reward models aim to infer task progress from visual observations, especially when privileged task states are less accessible in the real world than in simulation. Yet progress is inherently task-specific and history-dependent, making a globally normalized score difficult to calibrate. Scalable supervision therefore often relies on frame positions in successful demonstrations, implicitly coupling progress with elapsed time and providing weak supervision for stagnation, regression, and recovery. In this work, we present RoboStep, a framework for constructing online robot rewards that explicitly tracks task progress step by step, using pretrained VLM semantics for step completion verification rather than global progress calibration. We use a Frame-VLM to decompose tasks and generate step-local dense reward functions, while a Gate-VLM decides when the reward advances to the next step and switches the active reward function. Across extensive simulated manipulation experiments, we systematically compare these two roles of VLMs under matched policy-learning settings. Combined with recent pipelines that reconstruct real-world scenes in simulation, RoboStep further enables policies trained with structured rewards in simulation to transfer back to corresponding real scenes. Overall, we show that robot learning benefits from tracking task progress step by step: for online reward, VLMs are better used as step-completion verifiers than as global scalar progress predictors.

Primary area:
reinforcement learning
Keywords:
Robot Manipulation, Reinforcement Learning, Progress Reward Modeling, Vision-Language Models, Real-to-Sim-to-Real

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wJ1pOpenReview ↗
Rating
2(+1)
Confidence
3(+1)
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce RoboStep, a hierarchical reward system consists of 2 VLMs, a GPT 5.5 frontier model split the task into stages and generate dense reward protocol inside the stage, a Qwen opensource VLM check each frame if the next stage is achieved. The experiments show that RoboStep helps manipulation policy training with RL.

Critical strengths

The method is a training-free reward system, utilizing frontier VLM capabilities.

Critical weaknesses
  1. The method of split the long horizon task into stages/subtask is not new, SARM [1] has showed that this design is helpful for reward modeling and down-stream policy learning. RoboStep seems build on the same principle but not compare with SARM.

  2. Though authors have designed to use different tiers of model, but inference a 9B VLM each frame (or every several frames) still seem expensive and unscalable. Also there is no comparison with the relatively smaller reward models on reward qualities.

  3. Relying on VLM solely for stage splitting and dense reward function generation is unreliable. Also many details are not clear for the examples in A-3, A-4, how does Frame-VLM generate such detail rules by only read the task instruction without viewing a task demo ep?

  4. The authors only use GPT & Qwen combination for 2 VLMs, as an ablation study should at least use 2 frontier models and 2 opensourced models, together as 4 combinations, showing that the ability comes from RoboStep system design rather than purely the model ability.

  5. The real experiment is from real-to-sim-to-real, not directly showing that the reward system can work on real robot eps.

  6. The Figure 1 and 2 are very low quality, the color, font size, and position make the text very unclear.

Reference: [1] Chen, Qianzhong, et al. "Sarm: Stage-aware reward modeling for long horizon robot manipulation." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

How you make sure the Frame-VLM generation quality? Have you had any failure case that it’s bad generation cause the task harder to learn from sparse reward?

7 comments
AuthorsOpenReview ↗

Response to Reviewer wJ1p ( Weakness 1 )

Thanks for these thoughtful suggestions! Our responses are as follows:

Weakness 1

The method of split the long horizon task into stages/subtask is not new, SARM [1] has showed that this design is helpful for reward modeling and down-stream policy learning. RoboStep seems build on the same principle but not compare with SARM.

  • We deeply value stepwise and subtask-aware reward modeling as an important and active research direction, where SARM stands out as a particularly insightful and inspiring early contribution.
  • We carefully studied SARM during our initial literature review and were particularly impressed by its accurate stage and within-stage progress estimation from annotated demonstrations, especially its remarkable real-world T-shirt folding results with RA-BC. These results provide compelling evidence of the value of explicit subtask information for robot learning. This is also why we included both SARM and SARM2 in our Related Work discussion (Sec. 2).
  1. Contribution Clarification

We would like to clarify that we do not intend to claim novelty in decomposing long-horizon tasks into subtasks or novelty in demonstrating the benefits of stage information for reward modeling and downstream policy learning.

  • Through our literature review, we had carefully considered existing work exemplified by SARM on learning stage/progress representations from trajectories, and work exemplified by Code-as-Reward on encoding stepwise structure into reward/policy learning (a more detailed comparison is also provided in Appendix G-1, Table 21)
  • Building on these two lines of work, we hope to further emphasize maintaining persistent step state through visual semantic verification, and to examine its roles in explicit reward routing, step-index policy context, and composition/recovery potential through controlled experiments.
  1. Experiments Clarification
  • RoboStep focuses on online policy learning without pretrained policy initialization, task-specific demonstrations or real-world RL. Therefore, we selected five visual-reward baselines from related work (SOLE-R1, LRM, RoboMeter, RoboReward, and TOPReward) whose reward models are described in their respective papers as applicable without task-specific demonstration adaptation, and evaluated them under a shared online PPO training protocol.

  • We greatly appreciate SARM's demonstration-supervised approach to challenging deformable manipulation tasks such as T-shirt folding, where its reward model is trained on 200 densely annotated and 500 sparsely annotated trajectories. We are concerned that directly evaluating SARM under our demonstration-free setting may not adequately reflect its intended strengths.

  1. Further Summary

We see encoding task structure into reward/policy learning and learning stage/progress representations from trajectories as potentially complementary directions:

  • The latter is particularly valuable for fine-grained data annotation and Reward-Aligned Behavior Cloning.
  • The former may help generate task-specific trajectories for adapting the latter. More broadly, works such as RoboMeter leverage simulation data from LIBERO, MetaWorld, FAILSafe, and RLBench-based RACER.
AuthorsOpenReview ↗

Response to Reviewer wJ1p ( Weakness 2/3 & Question )

Weakness 2

Though authors have designed to use different tiers of model, but inference a 9B VLM each frame (or every several frames) still seem expensive and unscalable. Also there is no comparison with the relatively smaller reward models on reward qualities.

Thank you for raising these important concerns about inference cost and reward quality!

  1. Inference Cost Clarification.

    We would like to note that Gate-VLM is not invoked at every frame. Frame-VLM generates reward functions offline, while Gate-VLM is queried only when transition candidates are triggered. Between queries, the persistent step state is maintained and dense rewards are computed without VLM inference (Appendix A-2, C-3).

  2. Model Scale and Reward Quality.

We fully understand the reviewer's concern about model scale and reward quality. For additional context, we note that several related visual-reward methods also use VLMs of comparable sizes:

MethodSOLE-R1RoboMeterRoboRewardTOPRewardLRMRoboStep
Model Size8B4B4B / 8B8B8B9B

We further discuss expanded smaller-model experiments along with our response to Weakness 4.

  • Reward Quality. We suggest that reward quality in our framework may be related to step-completion verification accuracy and downstream policy performance. We hope that our false-acceptance/false-rejection rate (FAR/FRR) analysis (Sec. 5.3) and PPO success rates across 75 manipulation tasks (Table 2) could offer some preliminary evidence in this regard.

Weakness 3 & Question

Relying on VLM solely for stage splitting and dense reward function generation is unreliable. Also many details are not clear for the examples in A-3, A-4, how does Frame-VLM generate such detail rules by only read the task instruction without viewing a task demo ep?

How you make sure the Frame-VLM generation quality? Have you had any failure case that it’s bad generation cause the task harder to learn from sparse reward?

Thank you for these thoughtful questions about reward generation and its reliability!

  1. Frame-VLM Input Clarification.

    We would like to clarify that Frame-VLM does not rely solely on the task instruction. As illustrated in Fig. 2(a), its initial inputs include the task description, available simulator state/API interfaces, and reward design constraints.

    Appendix A-3 provides the prompts used for reward generation and visual verification, while Appendix A-4 presents representative generated reward programs. We further provide per-task Frame-VLM inputs, generated step decompositions, and dense reward components for all 75 tasks in Appendix G-2 (Table 24).

  2. Related Work. Previous works in Programmatic and Structured Reward Design have demonstrated the potential of language models for reward generation and task decomposition, such as Text2Reward (ICLR 2024), Eureka (ICLR 2024), Code-as-Reward (ICML 2024) and PRM4RL (NeurIPS 2025).

  3. Frame-VLM Generation Quality. We fully agree that generated reward quality is important. We also recognize the increasingly strong capabilities of recent foundation models in generating executable reward code.

    • In our initial Meta-World reward compilation using GPT-5.5 (xhigh), 19/50 tasks already met the 50% validation-success threshold without further reward revision.
    • As detailed in Appendix A-1, we adopt validation-guided reward refinement, similar to iterative reward improvement in Eureka and Text2Reward. When PPO validation success is below 50%, Frame-VLM revises the previous stage/reward design using training diagnostics including success rate and failure modes. We allow up to 5 trials to match Eureka's default iteration count, and use the final accepted or 5th-round reward program for evaluation. The distribution of compilation rounds across our 75 tasks is:
BenchmarkRound 1Round 2Round 3Round 4Round 5Total
MetaWorld191187550
ManiSkill6232619
SoftGym110136

We sincerely hope these clarifications and additional statistics help address the reviewer's concerns about the inputs, generation process and reliability of Frame-VLM.

AuthorsOpenReview ↗

Response to Reviewer wJ1p ( Weakness 4/5/6 )

Weakness 4

The authors only use GPT & Qwen combination for 2 VLMs, as an ablation study should at least use 2 frontier models and 2 opensourced models, together as 4 combinations, showing that the ability comes from RoboStep system design rather than purely the model ability.

Thank you for this valuable suggestion on cross-model generalizability!

  1. System Design Clarification.

    We view VLM capability and system design as complementary rather than mutually exclusive. VLMs provide semantic understanding, while RoboStep organizes it into persistent step state, explicit reward routing, and step-conditioned policy learning.

    To examine the contribution of system design under fixed model capabilities, Table 1 uses the same Frame-VLM-generated reward components and independently varies stepwise/all-dense reward structure and step-index policy inputs. Table 2 further includes Gate-Rule as a reference to distinguish visual verification from the underlying stepwise framework.

  2. Cross-Model Experiments.

    We fully agree that cross-model evaluation is also a valuable way to examine robustness. We used a 10-task MetaWorld subset to additionally evaluate different Frame-VLM and Gate-VLM combinations. The mean downstream PPO success rates are:

Frame-VLM / Gate-VLMQwen3.5-9B (Main)Qwen3.5-27BQwen3-VL-8BQwen3-VL-4BQwen3-VL-32B
GPT-5.5 xhigh (Main)38.31%46.66%39.25%30.22%34.51%
GPT-6 Astra xhigh40.07%45.18%41.37%35.67%36.12%
Claude Fable 5.141.13%47.04%40.21%37.45%35.08%

These results suggest that RoboStep can support policy learning across different Frame-VLM and Gate-VLM combinations. We hope it provides some additional evidence that RoboStep is not restricted to the original GPT-5.5 + Qwen3.5-9B combination.

We sincerely thank the reviewer again for advice on this additional evaluation.

Weakness 5

The real experiment is from real-to-sim-to-real, not directly showing that the reward system can work on real robot eps.

Thank you for raising this distinction.

  1. Real-World Deployment Clarification.

RoboStep is designed for from-scratch PPO training in simulation, followed by real-world deployment without demonstrations or real-world policy updates. Recent advances in real-to-sim-to-real scene reconstruction make this a promising option. Importantly, Gate-VLM still operates on real RGB observations during deployment, maintaining the persistent step state through visual verification (Sec. 4.3, Appendix D).

  1. Different Real-World Learning Settings.

We carefully reviewed related visual-reward methods that explore different real-world policy-learning protocols:

MethodReal-World Policy Learning Setting
SARMDemonstration-supervised stage-aware reward modeling & reward-aligned BC fine-tuning of pretrained policy
RoboRewardBridgeData V2-pretrained diffusion policy, 20 warm-start rollouts, followed by online DSRL-SAC
VLACApproximately 100 teleoperated trajectories for policy initialization, followed by real-world PPO with additional human-guidance
ReWiNDDemonstration-based reward and policy pretraining, followed by real-world online RL with human-provided success signals
TOPRewardPolicy pretrained on 200 hours of data, followed by reward-weighted BC using 50 demonstrations per task

We greatly appreciate these complementary approaches, each with different supervision and real-world interaction requirements, while our focus is on exploring whether structured visual rewards can enable demonstration-free simulation training and direct real-world transfer.

Weakness 6

The Figure 1 and 2 are very low quality, the color, font size, and position make the text very unclear.

Thank you for your feedback on the readability of Figures 1 and 2.

As RoboStep involves several interconnected components and experimental factors, we invested considerable effort in illustrating their interfaces and interactions in Figures 1 and 2. We regret that the figures may not have appeared as clear as intended.

Following your suggestions, we have revised the font sizes and color contrast in the updated PDF. We sincerely hope these improvements help address your concerns.

Summary

We sincerely appreciate your time and thoughtful feedback throughout this review and would be happy to clarify any remaining questions. If our responses have helped address your concerns, we would be very grateful if you might consider updating your score and confidence!

Reviewer_tCeaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes RoboStep, an online reward framework that tracks task progress step by step through an explicit step state machine, instead of predicting a single global progress score. The authors first introduce an offline Frame-VLM to decompose each task into steps and produce, for each step, a dense reward program together with a natural-language description of its completion standard. Built on the standard PPO training process, the generated reward program serves as the dense reward by activating only the current step's reward function under a persistent step index, which is also given to the actor and critic as a one-hot context. Second, an online Gate-VLM is introduced to visually verify whether the current step is complete and to authorize the transition to the next step. The main contribution of this paper is the claim that VLMs are better used as step-completion verifiers than as global scalar progress predictors, which matches intuition, and the authors run a large number of experiments both in simulation (Meta-World, ManiSkill and SoftGym) and on real robots through a real-to-sim-to-real pipeline.

Critical strengths
  1. S1. Step-level verification is a natural use of VLMs. Verifying whether one step is finished is much easier for a VLM than calibrating a global progress score, so step-gated rewards should give better-shaped dense feedback. The explicit step state also makes progress auditable, since every transition is a discrete, checkable event rather than a drifting scalar.

  2. S2. The simulation evaluation is extensive. The authors evaluate on 75 tasks across Meta-World, ManiSkill, and SoftGym, with five reproduced global-progress VLM baselines plus native/sparse controls. Gate-VLM scores 49.0%, versus 34.7% for the benchmark-native dense reward and 11.8% for the best global-progress VLM baseline (Table 2). Additional ablations cover reward structure, gate cadence, gate errors, composition, and recovery.

Critical weaknesses
  1. W1. The benchmark-native dense reward baseline appears under-trained. The official Meta-World report (arXiv:1910.10897, Appendix B) verifies that every task is solvable by single-task RL algorithms, with success rates far above those reported in Table 2 and Figure 3. Take soccer as an example: the officially reported single-task success rate is almost 100% with PPO, while Figure 3 shows less than 0.2. This gap indicates that the official baseline may not be convincing. The official MT50 multi-task PPO result (35.4%) is close to RoboStep's 35.6%, but the two would not be directly comparable if RoboStep trains policies independently per task, as Table 9's 41–45-dimensional inputs suggest. The authors should clarify whether training is single or multi-task.

  2. W2. Unexpected failure on one of the easiest tasks, indicating that the limitations and mechanism of the proposed method are not fully discussed. According to Figure 3, PickCube's success rate is barely 0 (pure white), while the native dense reward reaches 100%. There is no discussion of when or why RoboStep would fail on such an easy task. This suggests that the underlying mechanism of the proposed method has not been sufficiently investigated.

Questions for authors
  1. Policy training protocol and budget. Are the policies in Table 2 trained independently per task, or jointly as one multi-task policy per benchmark? The paper never states this explicitly, and while Table 8 lists the PPO hyperparameters, the training budget itself is never reported. Could you state the protocol and provide per-task training budgets and learning curves for the native baseline and RoboStep? Without this information, it is difficult to assess whether the native baseline is a converged reference and how strongly Table 2 supports the claimed advantage.
  2. Inconsistent sim-to-real numbers. §4.3 states that the step-conditioned actor improves average success from 63.5% to 84.0% in simulation and from 35.8% to 61.7% in the real world, but Table 3 directly above that sentence and App. Table 14 report 25.3→36.8 (sim) and 16.7→25.0 (real), and App. D-5 describes the gain as +11.5/+8.3 points. Where do the numbers in the main text come from, and why are they inconsistent with Table 3 and Table 14?
  3. The PickCube failure. Why does Gate-VLM achieve only near 0% on ManiSkill PickCube, while native dense reward and Gate-Rule reach about 100% and 80%, respectively? Please provide a mechanistic analysis of this failure with experimental evidence or a theoretical guarantee specifying the conditions under which the step-gated reward preserves the intended task-progress ordering.
  4. Generality of the method. The current evidence comes from a single Frame-VLM (gpt-5.5) and a Gate-VLM sweep limited to Qwen-family checkpoints on two tasks (Table 20). Do the generated stepwise reward programs and the gating behavior transfer to other VLMs, or do they depend on the specific models and prompts used here? Additional results with a different VLM would substantially strengthen the claim that RoboStep is a general recipe.
4 comments
AuthorsOpenReview ↗

Response to Reviewer tCea (Weakness 1 & Question 1/2)

Thank you so much for reading our paper so carefully and thoroughly!

W1 & Q1: Per-task vs. multi-task policy training & training budgets and learning curves.

  1. Experiments Clarification We rechecked our experimental settings and confirmed that the Meta-World native dense-reward results we reproduced in our original Table 2 used the official V2 human-shaped reward under the MT50 setting, where a single task-conditioned policy was trained across 50 tasks.

For all the other entries in Table 2, we train a separate policy for each task, broadly consistent with the task-specific policy-learning settings in the simulation used in related work (e.g., SOLE-R1 and RoboReward).

  1. Contribution Clarification We fully appreciate the importance of carefully human-shaped and rigorously validated benchmark-native rewards. Our intention is not to claim that our automatically generated rewards outperform these carefully designed and validated rewards. What we primarily hope to explore is:

    • the feasibility of using VLMs to visually verify subtask completion for online reward updates,

    • to ablate how explicit step information can support policy learning through state-machine reward routing and one-hot step-index inputs. We hope that including benchmark-native reward results can provide a useful reference for task solvability.

  2. Summary We sincerely thank the reviewer again for highlighting this important distinction!

  • We noticed that Meta-World+ (Appendix E.2) reports single-task PPO performance with a 20M-step training budget (approximately 90% success on at least 45/50 tasks) without specifying individual task convergence budgets.

  • For more clear presentation and avoiding a training-budget mismatch with the official results, we now use the independently trained single-task scores reported in the original Meta-World paper as a reference for task solvability in Table 2 and Figure 3. We have updated the PDF accordingly.

Q2: Typo in the Main-Text Results

Thank you for kindly pointing out this typo. We have carefully rechecked the results in Tables 3 and 14 and confirmed that they are correct. The typo in the main text has been corrected in the updated PDF.

AuthorsOpenReview ↗

Response to Reviewer tCea (Weakness 2 & Question 3/4 )

W2 & Q3: PickCube failure analysis, underlying mechanisms, and conditions for reliable stepwise rewards.

Thank you for highlighting this failure case! We revisited the original checkpoint and conducted additional trajectory diagnostics.

  1. Failure Case Analysis.

    • Observed failure. All failed episodes in this Gate-VLM case entered the transport stage while grasping the cube, and 89% reached the target but failed to satisfy the official robot-staticness condition.
    • Gate verification. Replacing Gate-VLM with Gate-Rule for the same frozen checkpoint left success rate unchanged (11%), suggesting that inference-time Gate-VLM were not responsible for this particular failure.
    • Taken together, these observations suggest that this checkpoint policy learned to reach the target but did not reliably learn to stop as required by the official task-success criterion.
  2. Additional Experiments.

We additionally retrained Gate-VLM and Gate-Rule policies with three different independent seeds:

MethodGate-VLMGate-Rule
PickCube SR (3 seeds)87.0%90.67%

These results suggest that the original low performance does not represent an inherent failure of the framework.

  1. Limitation Clarification.

    • To maintain a fair reward-generation protocol, Frame-VLM does not access benchmark-native rewards or official success predicates during reward compilation. Consequently, terminal requirements not explicitly captured by the task description, such as strict robot staticness or arrival velocity, may be insufficiently represented in the generated reward. We suppose that this can lead to discrepancies between shaped progress and official task success.

    • We have carefully investigated several mechanisms of RoboStep through controlled ablations: reward routing and step-index inputs (Sec. 4.1), Gate-VLM invocation cadence (Sec. 5.1), actor/critic step context (Sec. 5.2), and gate-error sensitivity (Sec. 5.3). We hope these studies could help clarify how stepwise information affects policy learning. The PickCube case further highlights an additional limitation: terminal-condition mismatch may also contribute to unexpected performance degradation.

We sincerely appreciate this question which helped us identify and better understand this specific failure mechanism!

Q4: Cross-model generalizability of Frame-VLM reward generation and Gate-VLM verification.

Thank you very much for this valuable suggestion on cross-model generalizability!

  1. Regarding prompts, we use simple, reusable templates without task-specific manual reward engineering with task descriptions and available state/API information provided as inputs (Appendix A-3).

  2. Additional Transfer Experiments. We fully agree that cross-model evaluation is a valuable way to prove robustness beyond specific models and prompts. Following this suggestion, we additionally evaluated different Frame-VLM and Gate-VLM combinations on a 10-task MetaWorld subset. The mean downstream PPO success rates (%) are:

Frame-VLM / Gate-VLMQwen3.5-9B (Main)Qwen3.5-27BQwen3-VL-8BQwen3-VL-4BQwen3-VL-32B
GPT-5.5 xhigh (Main)38.31%46.66%39.25%30.22%34.51%
GPT-6 Astra xhigh40.07%45.18%41.37%35.67%36.12%
Claude Fable 5.141.13%47.04%40.21%37.45%35.08%

We hope these additional experiments provide some evidence that RoboStep is not restricted to the original GPT-5.5 + Qwen3.5-9B combination. The results also suggest two interesting observations: larger Gate-VLM do not necessarily perform better, and different capable Frame-VLMs yield relatively similar downstream performance.

Summary

We sincerely thank the reviewer again for suggestions on these additional evaluations and we would be glad to address any further questions! If you find our additional experiments and clarifications satisfactory, we would sincerely appreciate your consideration of an updated score and confidence.

I thank the authors for their thorough responses and additional experiments, which have substantially addressed several of my concerns. In particular, I now accept the official single task Meta-World results as an appropriate reference for task solvability.

Nevertheless, I still have some concerns about the randomness. The PickCube results leave some open questions about the robustness of downstream policy training, given the substantial difference between the original low performance and the improved results obtained after retraining the policies with three independent seeds. The proposed terminal-condition mismatch is a plausible explanation. Further improving the stability of the method may require explicitly addressing this mismatch.

The additional cross-model generalization experiments are particularly interesting. I encourage the authors to further analyze why larger VLMs do not necessarily perform better, what mechanisms may underlie this observation, and what types of VLMs are better suited to this framework.

Overall, the responses and revisions have improved my assessment of the paper, and I have raised my rating accordingly.

AuthorsOpenReview ↗

Thanks to Reviewer tCea

Dear Reviewer tCea,

Thank you for your follow-up suggestions and for updating your assessment! We sincerely appreciate the time and care you have devoted to this discussion.

Your feedback has helped us better understand the importance of training stability and terminal-condition alignment. In future work, we plan to conduct more detailed failure analyses of the trajectories and policy checkpoints produced by our framework, and investigate which VLM capabilities best support stepwise reward generation and verification.

Thank you again for helping us strengthen both the paper and our understanding of the framework!

Best wishes,
The RoboStep Team

Reviewer_WGxXOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboStep represents robot-task progress as persistent discrete steps. An offline Frame-VLM generates step-local rewards, while an online Gate-VLM verifies transitions. The idea that VLMs may work better as transition verifiers than as global progress predictor is interesting.

Critical strengths
  1. Clear motivation for replacing ambiguous global progress scores with explicit step states.
  2. Useful separation between offline reward generation and online transition verification.
  3. Broad evaluation across 75 simulated tasks, several benchmarks, ablations, recovery, composition, and sim-to-real transfer.
  4. Explicit step state improves interpretability and supports rollback after regressions.
  5. Prompts, model choices, limitations, and AI use are disclosed. Overall, the core idea is promising and the empirical scope is ambitious.
Critical weaknesses
  1. The main simulation system still uses privileged state predicates to decide when to query Gate-VLM, so it is not fully visually driven
  2. Comparisons use different VLMs, query rates, and temporal adapters, making it difficult to isolate the benefit of step verification.
  3. Composition experiments train separate policies with budgets and horizons that grow with task length, rather than demonstrating unseen compositional generalization.
  4. Real-world evaluation covers only four tasks, has relatively low absolute success, and lacks visual-reward baselines.
Questions for authors

(1) How do the sim-to-real improvements reported in Section 4.3—63.5% to 84.0% and 35.8% to 61.7%—relate to Tables 3 and 14, which report 25.3% to 36.8% and 16.7% to 25.0%? (2) Does 99.23 ± 7.51 in Equation 34 suggest the task is almost trivial?

3 comments
AuthorsOpenReview ↗

Response to Reviewer WGxX (Weakness 1/2)

Thank you very much for your recognition of our work and for your valuable suggestions! Our responses are as follows:

Weakness 1

The main simulation system still uses privileged state predicates to decide when to query Gate-VLM, so it is not fully visually driven

Experimental Clarification. We also considered this when designing experiments:

  1. In Section 5.1, we investigate replacing the privilege-triggered Gate-VLM invocation with periodic queries at fixed intervals. Here, interval 64 means querying Gate-VLM every 64 Meta-World environment steps.

We put Table 6 below for convenient reference:

Table 6: Gate-VLM invocation cadence.

BenchmarkCadenceIntervalSuccess Rate
Meta-WorldFrequent (fast)1612.73 ± 1.46
Meta-WorldFrequent (mid)6423.60 ± 2.16
Meta-WorldFrequent (slow)12831.40 ± 3.34
ManiSkillFrequent (fast)831.25 ± 3.66
ManiSkillFrequent (mid)1643.67 ± 3.18
ManiSkillFrequent (slow)3238.33 ± 1.20
SoftGymFrequent (fast)828.56 ± 5.74
SoftGymFrequent (mid)1627.11 ± 5.02
SoftGymFrequent (slow)3228.22 ± 2.34

The detailed formulation of this ablation is provided in Appendix F-2:

χtfreq=1[zt<K] 1[t mod Fb=0]\chi_t^{\mathrm{freq}}=\mathbf{1}[z_t<K]\,\mathbf{1}[t\bmod F_b=0]

  1. In Section 5.2, we examine whether policy learning itself can be visually driven in the main Table 7:

Table 7: Policy-context ablations (%). ztz_t: Step index one-hot vector.

Actor inputActor ztz_tCritic ztz_tSR
State obs.✗✓67.20 ± 10.83
State obs.✗✗58.67 ± 2.49
DINOv2 + proprio✓✓37.20 ± 1.06
DINOv2 + proprio✗✓37.07 ± 0.96

Although removing privileged information leads to some degradation, the framework is still functional when turning visual driven.

We hope that our simulation results could serve as a preliminary demonstration of the framework's potential. We note, merely as additional context, that several related works primarily evaluate policy learning in simulation with privileged information and reward signals, such as Eureka (ICLR 2024), Code as Reward (ICML 2024), ARM-FM (ICLR 2026), and PRM4RL (NeurIPS 2025).

Weakness 2

Comparisons use different VLMs, query rates, and temporal adapters, making it difficult to isolate the benefit of step verification.

Thanks for pointing this important consideration for fair comparison! Here, we further clarify the parameter choices reported in Appendix C-2.

Baseline Setup. We use the released checkpoints of the original methods. For query rates and temporal adapters, we follow the original papers or refer to released PPO implementations:

MethodVLM Model Used (1)Query RatesTemporal Adapters and Output (4)
SOLE-R1 (NeurIPS 2026)Qwen3-VL-8B → SOLE-R1-8Bq=64 (2)Linear interpolation; Absolute task progress [-100, 100]
Large Reward Model (LRM) (arXiv 2026)Qwen3-VL-8B → LRM-progressq=10 (3)Reward holding; Absolute task progress [0, 1]
RoboMeter (RSS 2026)Qwen3-VL-4B → RoboMeter-4Bq=10 (3)Reward holding; Absolute task progress [0, 1] (per-frame)
RoboReward (arXiv 2026)Qwen3-VL-8B → RoboReward-8Bq=10 (3)Reward holding; Discrete trajectory score (1–5)
TOPReward (CoRL 2026)Qwen3-VL-8B, frozenq=10 (3)Reward holding; Task-completion token probability

(1). These baselines are primarily based on Qwen3-VL-8B. We used to try Qwen3-VL-8B on all 75 tasks and later switched to the newer Qwen3.5-9B. We revisit previous results and summarize below:

MethodMeta-WorldManiSkillSoftGymAvg.
Ours (Qwen3.5-9B)53.5%40.3%38.3%49.0%
Ours (Qwen3-VL-8B)54.9%36.9%32.4%48.54%

(2) We did not find a released PPO adapter for SOLE-R1 and explored query intervals with probe task from Table 18:

Query Intervalq=64q=32q=20q=10q=5
SOLE-R18.10%6.62%11.26%7.98%1.32%

Shorter intervals incur higher reward model inference costs, so we chose q=64 as a practical trade-off.

(3)(4) We follow SOLE-R1 in using linear interpolation for low-frequency VLM predictions, and follow LRM official repository in adopting reward holding (reusing the latest reward until the next query) and using q=10 for online PPO. LRM repo also integrates RoboMeter, RoboReward, and TOPReward with q=10.

AuthorsOpenReview ↗

Response to Reviewer WGxX (Weakness 3/4 and Question 1/2)

Weakness 3 & Question 2

W3: Composition experiments train separate policies with budgets and horizons that grow with task length, rather than demonstrating unseen compositional generalization.

Q2: Does 99.23 ± 7.51 in Equation 34 suggest the task is almost trivial?

Experiments Clarify This is indeed an important consideration for thoroughly evaluating the composition of stepwise rewards!

  • Budgets. In early trials with fixed budgets, we observed that longer compositions required more training to converge. To explore the maximum composition length, we therefore increase training budgets to reduce the risk of undertraining.
  • Horizons. Since the Actor control frequency remains fixed, we also extend episode horizons to accommodate longer trajectories, just like RoboMeter in its two-stage long-horizon task.

In Sec. 4.4, we divide our composition experiments into homogeneous and heterogeneous:

Homogenous Difficulty. Given that all homogeneous are without intermediate resets, and that the non-stepwise control (w/ sequential goal switching, w/o stepwise reward routing or stage-index policy inputs) achieves zero success on Homon≥2Homo_{n\ge 2} (as reported in Sec. 4.4 and Appendix E-1), these results may suggest that composition remains challenging without stepwise framework.

We additionally tested Homo-Push (pushing an object from A to B). The results show a similar pattern: the non-stepwise dense control succeeds only on Homo1Homo_1.

TaskHomo5Homo6Homo7Homo8Homo9Homo10Homo11Homo12Homo13Homo14Homo15
Push91.6%90.1%93.7%85.3%82.6%89.3%80.2%79.9%75.7%80.4%72.8%
PnP100%100%100%100%100%91.8%95.6%97.1%97.0%97.3%71.6%

Heterogenous & Unseen Generalization We extend our evaluation to representative 3-task and 5-task heterogeneous compositions to examine policy learning under unseen generalization, and report the highest success rates from the best checkpoints:

TaskAABABCBACCABABCABABCBAACBACBCABCCABCA
Best SR100%100%99.3%95.5%98.8%90.2%89.6%99.1%84.7%

A = pick-place, B = push, C = reach × 3. Each composition is executed end-to-end by a single policy without intermediate resets.

Weakness 4

Real-world evaluation covers only four tasks, has relatively low absolute success, and lacks visual-reward baselines.

Thank you for your suggestions on our real-world transfer evaluation!

For real-world transfer, our system adopts a real-to-sim-to-real approach, and thanks to recent advances in sim-ready scene reconstruction, this framework without pre-training, demonstrations or real-world RL has become a feasible option.

This differs from the real-world settings of prior baselines:

  • SOLE-R1 uses real-world online RL with DrQv2/SAC.
  • LRM starts from an SFT policy and collects 60 real-world rollouts per task with reward-weighted BC.
  • RoboMeter uses DSRL to steer a frozen DROID-pretrained π0\pi_0 checkpoint through real-world online RL.
  • RoboReward uses DSRL to refine a BridgeData V2-pretrained diffusion policy.
  • TOPReward also performs reward-weighted BC using 50 demonstrations per task.

We recognize that evaluating these baselines without pretrained policy initialization, demonstrations or real-world RL may not fully reflect their strengths.

For real-world tasks, SOLE-R1 is relatively closest to our setting as it supports policy learning from scratch without demonstrations. Following your suggestion, we additionally adapt SOLE-R1 under our real-to-sim-to-real pipeline as a reference comparison:

TaskOurs (Sim)Ours (Real)SOLE-R1 (Sim)SOLE-R1 (Real)
1.145.0%33.33%32.0%23.33%
1.225.0%13.33%12.0%3.33%
2.136.0%23.33%15.0%10.00%
2.241.0%30.00%22.0%16.67%

The results suggest that our method achieves better sim-to-real transfer under the same setting of no demonstrations or real-world RL.

Question 1

Thank you for catching this typo! We have rechecked the original data and confirmed that the results in Tables 3 and 14 are correct. The typo has been corrected in the updated PDF.

Summary

Thank you again for taking the time to share these valuable suggestions! If you have any further questions, please feel free to let us know and we will do our best to respond. If you feel that our responses have addressed your concerns, we would be truly grateful if you could consider updating your score and confidence.

Thank you for your response, clarification and the additional experiments. The comparison to different VLM and the extra results on real-world experiments demonstrates the robustness of the proposed method. I therefore maintain my positive scores.

Comments on the paper

1 comment

I. Acknowledgments

We sincerely thank all reviewers for their thoughtful feedback and constructive engagement throughout the discussion. Their suggestions have helped us strengthen the experimental evidence and clarify the contribution and scope of RoboStep.

Before the discussion\color{orange}{\text{Before the discussion}}, we appreciate the recognition from all three reviewers: WGxX described our core idea as “promising” and empirical scope as “ambitious”; tCea noted that “Step-level verification is a natural use of VLMs.” and “The simulation evaluation is extensive.”; and wJ1p highlighted that “The method is a training-free reward system, utilizing frontier VLM capabilities.”

During the discussion\color{orange}{\text{During the discussion}}, we were glad that our additional clarifications and experiments addressed concerns raised by the reviewers (e.g., “have substantially addressed several of my concerns” from Reviewer tCea). Reviewer WGxX also recognized the robustness demonstrated by the additional experiments, stating “I therefore maintain my positive scores.” We especially appreciate Reviewers tCea and wJ1p raising their ratings following the discussion.

II. Key Strengths

Reviewers highlighted the following strengths:

  • Motivation and VLM Semantics

    • Clear motivation for replacing ambiguous global progress scores with explicit step states. (WGxX)
    • Step-completion verification is a natural use of VLMs, avoiding the difficulty of global progress calibration. (tCea)
  • Structured Reward Design

    • Useful separation between offline reward generation and online transition verification. (WGxX)
    • Training-free reward system utilizing pretrained foundation-model capabilities. (wJ1p)
  • Interpretability and Recovery

    • Explicit step states make task progress auditable through discrete, checkable transitions. (tCea)
    • Interpretable progress tracking supports rollback after regressions. (WGxX)
  • Extensive Empirical Evaluation

    • Broad evaluation across 75 tasks spanning Meta-World, ManiSkill, and SoftGym. (WGxX, tCea)
    • Comparisons with five visual-reward baselines and benchmark-native/sparse controls. (tCea)
    • Extensive ablations covering reward structure, gate cadence, and gate errors, alongside composition and recovery experiments. (WGxX, tCea)
    • Evaluation of sim-to-real transfer in addition to simulation benchmarks. (WGxX)
  • Transparency

    • Disclosure of prompts, model choices and limitations. (WGxX)

III. Key Concerns and Our Responses

Key ConcernsReviewersOur Response
Contribution and prior work.wJ1pClarified persistent visual step tracking, reward routing, and policy conditioning, and distinguished our setting from SARM. W1 marked resolved.
Cross-model robustness.WGxX, tCea, wJ1pEvaluated 15 model combinations on 10 tasks and Qwen3-VL-8B across 75 tasks. Reviewers recognized the additional evidence; W4 marked resolved by wJ1p.
Native-reward reference.tCeaClarified training protocols and adopted official single-task Meta-World results as a task-solvability reference, explicitly accepted by the reviewer.
Real-world comparison.WGxXAdded SOLE-R1 under the same real-to-sim-to-real pipeline. RoboStep outperformed it on all four real-world tasks, with the evidence recognized by WGxX.
Visual rewards in simulation.wJ1pThe 5-seed SoftGym results showing RoboMeter/RoboReward outperforming native dense rewards on two deformable/fluid tasks, plus mean learning curves. The reviewer acknowledged the motivation and raised their score.
Reward generation and cost.wJ1pClarified generation inputs, validation-guided refinement, and compilation statistics; explained offline reward generation and event-triggered VLM queries.
Long-horizon composition.WGxXClarified budgets/horizons; added 15-stage Homo-Push and nine heterogeneous compositions without intermediate resets.

IV. Commitment to Revision

We have revised the manuscript to clarify RoboStep's contribution and experimental scope, update the benchmark reference, improve figure readability, and add representative performance learning curves. The revised version highlights the scope clarifications in blue, and Sec. 6 restores the expanded discussion. We will ensure that the final manuscript consolidates the additional experiments and clarifications presented throughout the rebuttal.

We sincerely appreciate the reviewers' constructive engagement and suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.