- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a specification-guided framework designed to provide dense and interpretable online rewards for RL training of VLAs. It addresses sample inefficiency of sparse binary rewards and the instability and compute overhead of VLM-based evaluators. SPECRL translates task instructions into STL specifications over task-level atomic propositions offline using an LLM-assisted process with human review, and then evaluates online progress using either privileged simulator states or a visual grounding pipeline that fuses 2D detections with depth maps to track 3D geometries relative to the end-effector.
Critical strengths
- The logic of computing STL specifications offline seems to be well-motivated. This eliminates the typical hallucination and unreliability issues with LLM-as-a-judge frameworks while being more informative than binary rewards.
- They show measurable improvement on cases where sparse exploration struggles most (LIBERO-Goal and LIBERO-Long). This is impressive since it is without requiring manually engineered steps
Critical weaknesses
- Despite the abstract claiming an automated LLM-based framework, from Fig 1 it seems that there is a human review/revise step that is only weakly quantified in the text. Furthermore, this framework makes the strong assumption that the instruction can be decomposed into atomic propositions from a fixed library, which limits its applicability. This limitation is not discussed well in the paper.
- The overall performance gain of privileged SPECRL over binary RL is marginal, and basically no advantage on short horizon. These differences are visually exaggerated across Fig 4,5,7 by truncating axes to start at 65-70%. Furthermore, from fig 5b it appears that without custom engineering heuristics the performance difference from binary rewards vanishes.
Questions for authors
- What percentage of LLM-generated STL specifications required manual human intervention? What would the downstream performance be if the human review is omitted?
- In Fig 2, task robustness has significant zero-gradient flat regions. How does the policy derive a learning signal with this, and why was a continuous relaxation on explored?
- Why are error bands in the figures omitted? Additionally, why was training in Figure 5 halted at 100 steps while the binary baseline was still trending upward, whereas Figure 4 evaluated out to 400 steps?
- Fig 3 shows detector confidence dropping from >0.9 to <0.4 during gripper contact. If the target object is occluded, how does the monitor prevent reward discontinuities? Furthermore, since proposition satisfaction relies on end-effector centroid proximity rather than contact force or finger closure, how does the system prevent the agent from exploiting the reward by hovering without grasping?
- Were any physical robot policies fine-tuned and executed closed-loop using the approach, or were all real-world evaluations conducted strictly offline on pre-recorded demonstration logs?
- What was the peak runtime GPU VRAM consumption during RL rollouts when running the YOLOE, depth-unprojection, and geometric tracking pipeline alongside the VLA policy? How does that compare to other methods?
7 comments
Response-2
Q4: Reward Stability under Occlusion and Prevention of Reward Exploitation
A: We thank the reviewer for this insightful question.
Occlusion Handling. The confidence values in Figure 3 come from the segmentation model, while rewards are computed from geometric margins using masks, depth, and robot proprioception. A lower detection confidence does not necessarily lead to a lower reward. When the target becomes fully occluded, the monitor uses its last observable state for reward computation and updates the estimate when the object becomes visible again. This helps reduce abrupt reward changes due to temporary detection loss. Moreover, visual occlusion is an inherent challenge across all vision-based reward formulation methods.
Preventing Reward Exploitation. We agree with the reviewer that proximity alone could encourage hovering without grasping. To address this, SpecRL evaluates grasp satisfaction through three jointly required conditions. As detailed in Appendix C.2, the visual grasp margin is , combining object–gripper proximity, proprioceptive gripper closure, and relative co-movement. Hovering near an object therefore cannot satisfy the grasp condition without gripper closure and evidence that the object moves with the gripper.
Q5: Real-World Evaluation and Closed-Loop Fine-Tuning
A: We thank the reviewer for this question. Our originally reported real-world evaluations were conducted offline on pre-recorded demonstration trajectories, allowing us to compare reward-tracking accuracy across methods using the same physical executions. During the rebuttal period, we extended this evaluation to include closed-loop online RL and additional trajectory-level analyses.
Real-robot online RL. We conducted online RL experiments on two manipulation tasks using a Franka robot. Following a lightweight real-world RL framework [4], we freeze the pretrained VLA and train a residual policy whose outputs are added to the VLA's proposed actions. We train this policy using either binary rewards or SpecRL-V rewards, with the same budget of 40 online interaction episodes per task. Each resulting policy is evaluated over 25 real-robot trials per task.
| Task | Initial policy | RL w/ Binary | RL w/ SpecRL-V |
|---|---|---|---|
| Blue bowl to pink plate | 72% | 92% | 100% |
| Blue cube to pink plate | 80% | 96% | 100% |
Under the same interaction budget, SpecRL-V achieves 100% success on both tasks, outperforming binary rewards by 8 and 4 percentage points, respectively. These results further validate the effectiveness of our framework in real-world online RL.
We have added a subsection on real-robot online RL experiments to Section 4.5 of the revised manuscript.
Expanded real-world trajectory evaluation. We also expanded the offline evaluation to 40 real-world trajectories across five manipulation tasks, covering successful and failed executions as well as novel object layouts. Alongside additional qualitative examples, we quantitatively evaluated how accurately each method tracks task progress using the Kendall and VOC metrics:
| Metric | SpecRL-V | Robo-Dopamine | TOPReward | RoboMeter |
|---|---|---|---|---|
| Kendall | 0.957 | 0.592 | 0.540 | 0.337 |
| VOC | 0.943 | 0.509 | 0.576 | 0.354 |
SpecRL-V achieves the highest scores on both metrics, with a Kendall score of 0.957 and a VOC of 0.943, indicating strong alignment between its reward estimates and task-progress ordering.
We have included both the additional qualitative visualizations and quantitative results in Appendix B of the revised manuscript.
[4] Zhao W, Yan X, Wan Z, et al. BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models[J]. arXiv preprint arXiv:2609.27450, 2026.
Q6: Peak VRAM Consumption
A: We thank the reviewer for asking about runtime memory usage. During RL rollouts, each GPU loads one reward model. We measured the peak VRAM used by the reward-computation pipeline on a single GPU, excluding the VLA policy:
| Reward Evaluator | SpecRL-V | Robo-Dopamine | RoboMeter | TOPReward |
|---|---|---|---|---|
| Peak VRAM (GiB) | 2.53 | 10.07 | 17.56 | 18.63 |
SpecRL-V uses 2.53 GiB, compared with 10.07-18.63 GiB for the other evaluators, reducing reward-computation memory usage by approximately 75-86%.
We have included these results in Appendix B of the revised manuscript.
Response-3
W1: Applicability of the Atomic Proposition Library
A: We thank the reviewer for this question. While our current implementation uses seven atomic propositions that successfully cover all evaluated LIBERO tasks through composition, this vocabulary is extensible. When a new task introduces an operation beyond the existing library, a new proposition and its corresponding grounding function can be straightforwardly defined and added.
We have added a discussion to the revised manuscript to clarify this point.
W2: Performance Gains across Task Horizons
A: We thank the reviewer for this observation. The more modest gains observed on short-horizon tasks can be attributed to the high performance of the initial policy. For example, as shown in Figure 4, the initial policy's success rate on LIBERO-Object is 89.2%, meaning that successful trajectories are encountered relatively frequently during exploration. Binary rewards can therefore provide useful learning signals, and both methods achieve high success rates after fine-tuning: 97.0% for binary rewards and 98.2% for SpecRL-P.
SpecRL's intermediate feedback is particularly useful for longer-horizon tasks with lower initial success rates, where achieving full task success during exploration is more difficult. As shown in Figure 4, SpecRL-P improves success rates over binary rewards by 7.2 and 4.8 percentage points on LIBERO-Goal and LIBERO-Long, respectively.
W2: Axis Ranges in the Learning Curves
A: We thank the reviewer for this question. We chose this y-axis range because our RL fine-tuning starts from pre-trained VLA checkpoints with a baseline success rate of around 70%, rather than starting from 0%. Starting the y-axis at 0% would leave a large blank area containing no meaningful data, making the performance differences during RL fine-tuning less visible.
W2: Contribution of Reward Design Components
A: We thank the reviewer for this observation. SpecRL w/o Norm & Stage indeed performs similarly to binary rewards in Figure 5b. This variant removes normalization and stage monitoring, which can mask intermediate progress and distort the relative contributions of different margins. As a consequence, its feedback may provide limited additional guidance over binary rewards. This observation highlights the importance of our full reward design in translating specifications into more informative guidance for policy learning.
We sincerely thank the reviewer again for the insightful comments. If you have any further questions or additional concerns, please feel free to discuss them with us.
Response-1
We sincerely thank Reviewer bZkx for the careful reading and constructive feedback. We appreciate your recognition of SpecRL's offline STL specification design and its performance gains on challenging long-horizon tasks. Below, we provide detailed responses to your concerns.
Q1 & W1: Human Intervention
A: We thank the reviewer for raising this question. We used the GPT-4o API to generate specifications for all 40 LIBERO tasks. Two human experts reviewed their object references, stage ordering, and consistency with the task environment. Only two specifications (5%) required correction. The review times across the benchmark suites are detailed below:
The review required 24.21 minutes in total, averaging approximately 36 seconds per task. It is performed once offline, after which the validated specifications are cached and reused throughout RL fine-tuning without further review.
Compared to the heavy computational cost of RL training (e.g., nearly 4 days per training run on an eight-A100 server for 400 epochs), this offline review overhead is negligible. Consequently, we did not perform an ablation comparing SpecRL with uncorrected specifications, because such unvalidated specifications would introduce unnecessary noise, whereas our verification requires minimal effort to guarantee correct reward shaping.
We have added the review times to Appendix C.1.
Q2: Task Robustness and Learning Signal
A: We thank the reviewer for this question. PPO updates the policy using sampled returns and advantage estimates, rather than backpropagating through the reward function. Thus, a zero derivative of robustness with respect to the state does not necessarily imply a zero policy gradient. For example, binary success rewards also remain zero before task completion, yet successful rollouts can provide a learning signal through their returns. The practical challenge is that flat reward regions provide limited feedback about intermediate progress.
To clarify, Figure 2 was intended to illustrate the limitations of standard min-max STL robustness, rather than the final reward used by SpecRL. As the reviewer accurately observed, the standard minimum operator masks progress and creates those flat regions. This specific limitation is precisely what motivated our reward construction in Section 3.3. SpecRL combines AGM aggregation with stage monitoring to provide more informative feedback within and across stages. As illustrated in Figure 6, the reward curves on real-world robot trajectories capture intermediate progress without the flat regions shown in Figure 2.
Q3: Error Bands and Training Budgets
A: We thank the reviewer for allowing us to clarify our experimental protocol.
Error Bands: As noted in Appendix A, we follow the common practice in computationally intensive large-scale model training by using a fixed random seed [1-3], primarily due to the prohibitive computational cost. For example, our available infrastructure consists of a single server with eight NVIDIA A100 GPUs, where a single 400-epoch RL training run requires nearly four days; running multiple seeds would demand weeks of continuous computation.
Training Budgets: Figures 4 and 5 serve different experimental purposes and utilize different training budgets. Figure 4 presents our main evaluation, where SpecRL-P is compared against the binary-reward baseline over 400 PPO updates across all LIBERO suites. In contrast, Figure 5 compares various reward formulations and ablated variants under a common budget of 100 updates on LIBERO-Goal. In line with standard practice in recent studies [3], we adopted this 100-update limit for ablations and comparisons due to severe computational constraints. For instance, a single 100-update training run for a baseline method such as Robo-Dopamine or RoboMeter requires nearly three days, meaning a complete 400-epoch run would take around 12 days. We observed that 100 epochs were already sufficient to clearly reveal performance trends and relative differences among the ablated variants and other reward baselines.
[1] Li H, Zuo Y, Yu J, et al. Simplevla-rl: Scaling vla training via reinforcement learning[C]//International Conference on Learning Representations. 2026: 125846-125869.
[2] Lu G, Guo W, Zhang C, et al. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning[J]. arXiv preprint arXiv:2505.18719, 2025.
[3] Chen K, Liu Z, Zhang T, et al. πrl: Online rl fine-tuning for flow-based vision-language-action models[J]. arXiv preprint arXiv:2510.25889, 2025.