acceptodds
Under review as a conference paper at ICLR 2027

SpecRL: Specification-Guided Dense Rewards for VLA Reinforcement Learning

Abstract

Reinforcement learning fine-tuning (RLFT) has emerged as a promising approach for adapting pretrained vision-language-action models through task-directed interaction. However, many existing methods rely on sparse binary rewards, which provide limited guidance for policy optimization and thereby hinder learning performance. While recent approaches seek denser feedback via LLM-generated reward functions or vision-language model evaluations, they are often inefficient and the resulting rewards are often unreliable, potentially introducing errors that misguide policy optimization. In this work, we propose SpecRL, a specification-guided framework designed to provide dense, interpretable online rewards for RLFT. The core idea is to formalize task requirements in Signal Temporal Logic (STL) and derive rewards from its explicit quantitative semantics. SpecRL first uses an LLM-aided offline process to convert language instructions into STL specifications constituted by task-level atomic propositions. During execution, SpecRL evaluates proposition satisfaction using either a privileged-state grounding module or a lightweight visual grounding module, supporting both simulation and real-world deployments. A runtime monitor then aggregates the quantitative values of propositions along the specification's temporal structure, translating fine-grained task progress into dense rewards. Extensive evaluations on the LIBERO benchmark show that privileged-state SpecRL improves the success rate over binary-reward RLFT by percentage points, while visual SpecRL surpasses the strongest visual baseline by percentage points with faster training. Real-world trajectory evaluations further demonstrate that its deterministic and interpretable design delivers more accurate reward feedback than baselines.

Primary area:
reinforcement learning
Keywords:
Vision-Language-Action Models, Reinforcement Learning Fine-Tuning, Signal Temporal Logic, Specification-Guided Rewards

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bZkxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a specification-guided framework designed to provide dense and interpretable online rewards for RL training of VLAs. It addresses sample inefficiency of sparse binary rewards and the instability and compute overhead of VLM-based evaluators. SPECRL translates task instructions into STL specifications over task-level atomic propositions offline using an LLM-assisted process with human review, and then evaluates online progress using either privileged simulator states or a visual grounding pipeline that fuses 2D detections with depth maps to track 3D geometries relative to the end-effector.

Critical strengths
  1. The logic of computing STL specifications offline seems to be well-motivated. This eliminates the typical hallucination and unreliability issues with LLM-as-a-judge frameworks while being more informative than binary rewards.
  2. They show measurable improvement on cases where sparse exploration struggles most (LIBERO-Goal and LIBERO-Long). This is impressive since it is without requiring manually engineered steps
Critical weaknesses
  1. Despite the abstract claiming an automated LLM-based framework, from Fig 1 it seems that there is a human review/revise step that is only weakly quantified in the text. Furthermore, this framework makes the strong assumption that the instruction can be decomposed into atomic propositions from a fixed library, which limits its applicability. This limitation is not discussed well in the paper.
  2. The overall performance gain of privileged SPECRL over binary RL is marginal, and basically no advantage on short horizon. These differences are visually exaggerated across Fig 4,5,7 by truncating axes to start at 65-70%. Furthermore, from fig 5b it appears that without custom engineering heuristics the performance difference from binary rewards vanishes.
Questions for authors
  1. What percentage of LLM-generated STL specifications required manual human intervention? What would the downstream performance be if the human review is omitted?
  2. In Fig 2, task robustness has significant zero-gradient flat regions. How does the policy derive a learning signal with this, and why was a continuous relaxation on explored?
  3. Why are error bands in the figures omitted? Additionally, why was training in Figure 5 halted at 100 steps while the binary baseline was still trending upward, whereas Figure 4 evaluated out to 400 steps?
  4. Fig 3 shows detector confidence dropping from >0.9 to <0.4 during gripper contact. If the target object is occluded, how does the monitor prevent reward discontinuities? Furthermore, since proposition satisfaction relies on end-effector centroid proximity rather than contact force or finger closure, how does the system prevent the agent from exploiting the reward by hovering without grasping?
  5. Were any physical robot policies fine-tuned and executed closed-loop using the approach, or were all real-world evaluations conducted strictly offline on pre-recorded demonstration logs?
  6. What was the peak runtime GPU VRAM consumption during RL rollouts when running the YOLOE, depth-unprojection, and geometric tracking pipeline alongside the VLA policy? How does that compare to other methods?
7 comments
AuthorsOpenReview ↗

Response-1

We sincerely thank Reviewer bZkx for the careful reading and constructive feedback. We appreciate your recognition of SpecRL's offline STL specification design and its performance gains on challenging long-horizon tasks. Below, we provide detailed responses to your concerns.

Q1 & W1: Human Intervention

A: We thank the reviewer for raising this question. We used the GPT-4o API to generate specifications for all 40 LIBERO tasks. Two human experts reviewed their object references, stage ordering, and consistency with the task environment. Only two specifications (5%) required correction. The review times across the benchmark suites are detailed below:

MetricLIBERO-SpatialLIBERO-ObjectLIBERO-GoalLIBERO-Long
Total Review Time (min)5.235.925.617.45

The review required 24.21 minutes in total, averaging approximately 36 seconds per task. It is performed once offline, after which the validated specifications are cached and reused throughout RL fine-tuning without further review.

Compared to the heavy computational cost of RL training (e.g., nearly 4 days per training run on an eight-A100 server for 400 epochs), this offline review overhead is negligible. Consequently, we did not perform an ablation comparing SpecRL with uncorrected specifications, because such unvalidated specifications would introduce unnecessary noise, whereas our verification requires minimal effort to guarantee correct reward shaping.

We have added the review times to Appendix C.1.

Q2: Task Robustness and Learning Signal

A: We thank the reviewer for this question. PPO updates the policy using sampled returns and advantage estimates, rather than backpropagating through the reward function. Thus, a zero derivative of robustness with respect to the state does not necessarily imply a zero policy gradient. For example, binary success rewards also remain zero before task completion, yet successful rollouts can provide a learning signal through their returns. The practical challenge is that flat reward regions provide limited feedback about intermediate progress.

To clarify, Figure 2 was intended to illustrate the limitations of standard min-max STL robustness, rather than the final reward used by SpecRL. As the reviewer accurately observed, the standard minimum operator masks progress and creates those flat regions. This specific limitation is precisely what motivated our reward construction in Section 3.3. SpecRL combines AGM aggregation with stage monitoring to provide more informative feedback within and across stages. As illustrated in Figure 6, the reward curves on real-world robot trajectories capture intermediate progress without the flat regions shown in Figure 2.

Q3: Error Bands and Training Budgets

A: We thank the reviewer for allowing us to clarify our experimental protocol.

Error Bands: As noted in Appendix A, we follow the common practice in computationally intensive large-scale model training by using a fixed random seed [1-3], primarily due to the prohibitive computational cost. For example, our available infrastructure consists of a single server with eight NVIDIA A100 GPUs, where a single 400-epoch RL training run requires nearly four days; running multiple seeds would demand weeks of continuous computation.

Training Budgets: Figures 4 and 5 serve different experimental purposes and utilize different training budgets. Figure 4 presents our main evaluation, where SpecRL-P is compared against the binary-reward baseline over 400 PPO updates across all LIBERO suites. In contrast, Figure 5 compares various reward formulations and ablated variants under a common budget of 100 updates on LIBERO-Goal. In line with standard practice in recent studies [3], we adopted this 100-update limit for ablations and comparisons due to severe computational constraints. For instance, a single 100-update training run for a baseline method such as Robo-Dopamine or RoboMeter requires nearly three days, meaning a complete 400-epoch run would take around 12 days. We observed that 100 epochs were already sufficient to clearly reveal performance trends and relative differences among the ablated variants and other reward baselines.

[1] Li H, Zuo Y, Yu J, et al. Simplevla-rl: Scaling vla training via reinforcement learning[C]//International Conference on Learning Representations. 2026: 125846-125869.

[2] Lu G, Guo W, Zhang C, et al. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning[J]. arXiv preprint arXiv:2505.18719, 2025.

[3] Chen K, Liu Z, Zhang T, et al. πrl: Online rl fine-tuning for flow-based vision-language-action models[J]. arXiv preprint arXiv:2510.25889, 2025.

AuthorsOpenReview ↗

Response-2

Q4: Reward Stability under Occlusion and Prevention of Reward Exploitation

A: We thank the reviewer for this insightful question.

Occlusion Handling. The confidence values in Figure 3 come from the segmentation model, while rewards are computed from geometric margins using masks, depth, and robot proprioception. A lower detection confidence does not necessarily lead to a lower reward. When the target becomes fully occluded, the monitor uses its last observable state for reward computation and updates the estimate when the object becomes visible again. This helps reduce abrupt reward changes due to temporary detection loss. Moreover, visual occlusion is an inherent challenge across all vision-based reward formulation methods.

Preventing Reward Exploitation. We agree with the reviewer that proximity alone could encourage hovering without grasping. To address this, SpecRL evaluates grasp satisfaction through three jointly required conditions. As detailed in Appendix C.2, the visual grasp margin is mpickvis=AGM⁡(mnear,mclose,mcomove)m_{\mathrm{pick}}^{\mathrm{vis}}=\operatorname{AGM}(m_{\mathrm{near}},m_{\mathrm{close}},m_{\mathrm{comove}}), combining object–gripper proximity, proprioceptive gripper closure, and relative co-movement. Hovering near an object therefore cannot satisfy the grasp condition without gripper closure and evidence that the object moves with the gripper.

Q5: Real-World Evaluation and Closed-Loop Fine-Tuning

A: We thank the reviewer for this question. Our originally reported real-world evaluations were conducted offline on pre-recorded demonstration trajectories, allowing us to compare reward-tracking accuracy across methods using the same physical executions. During the rebuttal period, we extended this evaluation to include closed-loop online RL and additional trajectory-level analyses.

Real-robot online RL. We conducted online RL experiments on two manipulation tasks using a Franka robot. Following a lightweight real-world RL framework [4], we freeze the pretrained VLA and train a residual policy whose outputs are added to the VLA's proposed actions. We train this policy using either binary rewards or SpecRL-V rewards, with the same budget of 40 online interaction episodes per task. Each resulting policy is evaluated over 25 real-robot trials per task.

TaskInitial policyRL w/ BinaryRL w/ SpecRL-V
Blue bowl to pink plate72%92%100%
Blue cube to pink plate80%96%100%

Under the same interaction budget, SpecRL-V achieves 100% success on both tasks, outperforming binary rewards by 8 and 4 percentage points, respectively. These results further validate the effectiveness of our framework in real-world online RL.

We have added a subsection on real-robot online RL experiments to Section 4.5 of the revised manuscript.

Expanded real-world trajectory evaluation. We also expanded the offline evaluation to 40 real-world trajectories across five manipulation tasks, covering successful and failed executions as well as novel object layouts. Alongside additional qualitative examples, we quantitatively evaluated how accurately each method tracks task progress using the Kendall and VOC metrics:

MetricSpecRL-VRobo-DopamineTOPRewardRoboMeter
Kendall0.9570.5920.5400.337
VOC0.9430.5090.5760.354

SpecRL-V achieves the highest scores on both metrics, with a Kendall score of 0.957 and a VOC of 0.943, indicating strong alignment between its reward estimates and task-progress ordering.

We have included both the additional qualitative visualizations and quantitative results in Appendix B of the revised manuscript.

[4] Zhao W, Yan X, Wan Z, et al. BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models[J]. arXiv preprint arXiv:2609.27450, 2026.

Q6: Peak VRAM Consumption

A: We thank the reviewer for asking about runtime memory usage. During RL rollouts, each GPU loads one reward model. We measured the peak VRAM used by the reward-computation pipeline on a single GPU, excluding the VLA policy:

Reward EvaluatorSpecRL-VRobo-DopamineRoboMeterTOPReward
Peak VRAM (GiB)2.5310.0717.5618.63

SpecRL-V uses 2.53 GiB, compared with 10.07-18.63 GiB for the other evaluators, reducing reward-computation memory usage by approximately 75-86%.

We have included these results in Appendix B of the revised manuscript.

AuthorsOpenReview ↗

Response-3

W1: Applicability of the Atomic Proposition Library

A: We thank the reviewer for this question. While our current implementation uses seven atomic propositions that successfully cover all evaluated LIBERO tasks through composition, this vocabulary is extensible. When a new task introduces an operation beyond the existing library, a new proposition and its corresponding grounding function can be straightforwardly defined and added.

We have added a discussion to the revised manuscript to clarify this point.

W2: Performance Gains across Task Horizons

A: We thank the reviewer for this observation. The more modest gains observed on short-horizon tasks can be attributed to the high performance of the initial policy. For example, as shown in Figure 4, the initial policy's success rate on LIBERO-Object is 89.2%, meaning that successful trajectories are encountered relatively frequently during exploration. Binary rewards can therefore provide useful learning signals, and both methods achieve high success rates after fine-tuning: 97.0% for binary rewards and 98.2% for SpecRL-P.

SpecRL's intermediate feedback is particularly useful for longer-horizon tasks with lower initial success rates, where achieving full task success during exploration is more difficult. As shown in Figure 4, SpecRL-P improves success rates over binary rewards by 7.2 and 4.8 percentage points on LIBERO-Goal and LIBERO-Long, respectively.

W2: Axis Ranges in the Learning Curves

A: We thank the reviewer for this question. We chose this y-axis range because our RL fine-tuning starts from pre-trained VLA checkpoints with a baseline success rate of around 70%, rather than starting from 0%. Starting the y-axis at 0% would leave a large blank area containing no meaningful data, making the performance differences during RL fine-tuning less visible.

W2: Contribution of Reward Design Components

A: We thank the reviewer for this observation. SpecRL w/o Norm & Stage indeed performs similarly to binary rewards in Figure 5b. This variant removes normalization and stage monitoring, which can mask intermediate progress and distort the relative contributions of different margins. As a consequence, its feedback may provide limited additional guidance over binary rewards. This observation highlights the importance of our full reward design in translating specifications into more informative guidance for policy learning.


We sincerely thank the reviewer again for the insightful comments. If you have any further questions or additional concerns, please feel free to discuss them with us.

Reviewer_YthLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpecRL introduces dense rewards for vision-language-action model fine-tuning by using an LLM that translates task instructions into Signal Temporal Logic specifications for a one-time human review. At runtime, the system evaluates atomic propositions through simulator state or a lightweight visual module, while a stage-wise monitor transforms task progress into rewards. When tested on LIBERO using π0.5\pi_{0.5} and PPO, the mean success rates from 89.3% with binary rewards to 92.6%.

Critical strengths
  1. The visual variant uses a 44 MB segmentation model instead of a VLM evaluator, so it is very lightweight. It outperforms the baselines while training much faster. This is important because reward-model cost is a major bottleneck in online VLA RLFT.

  2. The reward design is interpretable and can be checked by a human, which makes it less of a black box. This is good because it addresses the problem of unreliable rewards reinforcing wrong behavior through a more inspectable reward design.

Critical weaknesses
  1. The specifications are just ordered sequences of subgoals. Richer STL features such as safety constraints or timing bounds are not used. Most of the reward signal comes from hand-designed margins, tuned thresholds, and human-corrected specifications. It is therefore unclear whether the improvements come from the specification framework or from manual reward, especially since Text2Reward receives no human correction.

  2. The paper claims real-world applicability, but all RL training is done in simulation. The only real-world result is qualitative reward curves on five demonstration trajectories, which cannot demonstrate generalizability. Extending the method to new tasks also requires new propositions, thresholds, and segmentation models. The applicability and generalizability of the framework is unclear, and also how much effort is needed when adapted to new tasks.

Questions for authors
  1. Can you please provide more qualitative results on real-world samples, with a larger sample size? The current 5 samples can not strongly show the ability of the framework.
  2. Can you provide more detail on how the propositions, thresholds, and segmentation models were obtained, and how much effort this required?
  3. See weakness 1. Can you further clarify how much of the improvement over Text2Reward comes from human correction of the specifications? How much effort did this correction require, e.g., how many of the LIBERO specifications needed revision?
8 comments
AuthorsOpenReview ↗

Response-1

We sincerely thank Reviewer YthL for the insightful comments. We are encouraged that you recognized the advantages of SpecRL in terms of interpretability and online efficiency. Detailed responses to your concerns are provided below.

Q1 & W2: Real-World Applicability and Qualitative Samples

A: We thank the reviewer for this suggestion. During the rebuttal period, we expanded our analyses to 40 real-world trajectories across five manipulation tasks, covering successful and failed executions as well as novel object layouts. We have included qualitative visualizations of selected representative trajectories in Appendix B of the revised manuscript.

To complement these examples, we quantitatively evaluated how accurately each method tracks task progress on the collected trajectories using the Kendall and VOC metrics:

MetricSpecRL-VRobo-DopamineTOPRewardRoboMeter
Kendall0.9570.5920.5400.337
VOC0.9430.5090.5760.354

SpecRL-V achieves the highest scores on both metrics, with a Kendall score of 0.957 and a VOC of 0.943, indicating strong alignment between its reward estimates and task-progress ordering. These results provide further evidence that SpecRL-V offers informative progress feedback across varied real-world executions. We have also included these quantitative results in Appendix B of the revised manuscript.

In addition, we have conducted real-robot online RL experiments on two manipulation tasks using a Franka robot. Following a lightweight real-world RL framework [1], we freeze the pretrained VLA and train a residual policy whose outputs are added to the VLA's proposed actions. We train this policy using either binary rewards or SpecRL-V rewards, with the same budget of 40 online interaction episodes per task. Each resulting policy is evaluated over 25 real-robot trials per task.

TaskInitial policyRL w/ BinaryRL w/ SpecRL-V
Blue bowl to pink plate72%92%100%
Blue cube to pink plate80%96%100%

Under the same interaction budget, SpecRL-V achieves 100% success on both tasks, outperforming binary rewards by 8 and 4 percentage points, respectively. These results further validate the effectiveness of our framework in real-world online RL.

We have added a subsection on real-robot online RL experiments to Section 4.5 of the revised manuscript.

[1] Zhao W, Yan X, Wan Z, et al. BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models[J]. arXiv preprint arXiv:2609.27450, 2026.

Q2 & W2: Proposition Design, Threshold Calibration, and Perception Adaptation

A: We thank the reviewer for this practical question. We summarize how these components were constructed and the associated setup effort below.

1. Proposition Design. We defined seven task-level atomic propositions to represent the manipulation operations and spatial relations required by the evaluated tasks. Each proposition is grounded through an explicit function that computes its satisfaction margin from simulator states or visual observations and robot proprioception. These reusable components cover all LIBERO tasks (LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long).

2. Threshold Calibration. We calibrated the thresholds using the state values relevant to each condition at successful completion in task environments. For the privileged variant, they were derived directly from simulator ground-truth states, requiring less than 1 hour of calibration. For the visual variant, calibration used the corresponding visual observations and required 2–3 hours to account for perception noise and depth variation. Once calibrated, these configurations are shared across tasks.

3. Segmentation Models. We fine-tuned separate segmentation models for simulation and real-world deployment. As reported in Appendix A, the simulation dataset contained 11,519 frames covering 26 classes, while the real-world dataset contained 220 demonstration frames covering four classes. Both models were fine-tuned for 80 epochs, requiring less than 1 hour of training in total.

Although SpecRL-V requires a one-time offline setup, it considerably reduces RL training time. As shown in Figure 7, over 100 epochs on a server with eight A100 GPUs, SpecRL-V reduces training time by 41.78, 12.26, and 39.13 hours compared to Robo-Dopamine, TOPReward, and RoboMeter, respectively, while using less GPU memory and achieving higher performance. Even the smallest reduction of 12.26 hours exceeded the time we spent on manual design. The time savings can be even greater in longer training runs.

We have added the offline setup details to Appendix C.

AuthorsOpenReview ↗

Response-2

Q3 & W1: Human Correction Effort and Comparison with Text2Reward

A: We thank the reviewer for this insightful question. First, we would like to clarify that Text2Reward similarly relies on expert-provided descriptions of environment states and APIs, and requires human-in-the-loop feedback to interactively refine reward functions.

In our framework, the human review step is introduced to ensure precision. Utilizing the GPT-4o API to generate initial specifications, out of all 40 LIBERO tasks, only 2 specifications (5%) required manual correction. The review times across the benchmark suites are detailed below:

MetricLIBERO-SpatialLIBERO-ObjectLIBERO-GoalLIBERO-Long
Total Review Time (min)5.235.925.617.45

Across all 40 tasks, the entire review process required only 24.21 minutes, averaging approximately 36 seconds per task. This validation is conducted once offline; the validated JSON specifications are cached and reused throughout RL fine-tuning without any further human intervention.

Compared to the multi-day computational cost of RL training (e.g., nearly 4 days per training run on an eight-A100 server for 400 epochs), this one-time offline review overhead is entirely negligible. Consequently, we did not perform an ablation comparing SpecRL with uncorrected specifications against Text2Reward, because such unvalidated specifications would introduce noise, whereas our verification requires minimal effort to guarantee correct reward shaping.

We have added the review times to Appendix C.1.

W1: STL Expressiveness Beyond Ordered Subgoals

A: We thank the reviewer for this insightful question. Our current specifications were designed around the task-completion requirements of the evaluated tasks. We use ordered subgoals without explicit timing bounds because these tasks do not prescribe completion windows for individual stages, whose durations vary during policy learning. The current LIBERO evaluation also focuses on task completion without additional safety constraints. Beyond representing subgoal order, SpecRL combines AGM robustness and finite-state stage monitoring to turn individual geometric margins into dense feedback reflecting progress within and across stages.

However, the STL formulation provides a natural basis for incorporating these requirements. For example, a task specification could be augmented with an ''always avoid collision'' condition. The existing YOLOE-based perception pipeline could support its grounding by distinguishing manipulated objects from surrounding objects and computing distance-based safety margins. Similarly, when a task specifies meaningful completion windows, bounded temporal operators could express these timing requirements.

We have added a discussion of these safety and timing extensions to the revised manuscript.


We sincerely thank the reviewer again for the insightful comments. If you have any further questions or additional concerns, please feel free to discuss them with us.

Dear Reviewer YthL,

Thank you again for taking the time to review our paper. Your insightful comments have helped us improve the paper.

With about a day and a half left in the discussion period, we would appreciate it if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to answer any further questions.

Best regards,

The Authors

Reviewer_Amr5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the reward design problem in reinforcement learning fine-tuning of vision-language-action (VLA) models: when rewards are provided only upon task success, robots receive little guidance on which intermediate actions are helpful. SpecRL first uses a large language model to translate natural-language instructions into Signal Temporal Logic (STL) specifications, followed by a one-time human review. For example, “put the bowl into the drawer and close it” can be decomposed into “pick up the bowl → place it inside the drawer → close the drawer.” During execution, the system uses either simulator states or object segmentation, depth maps, and robot proprioception to measure progress within each stage. It then combines these measurements with stage monitoring to generate stepwise rewards for policy optimization. Experiments show that the privileged-state version improves the average success rate on LIBERO from 89.3% to 92.6%. On LIBERO-Goal, the visual version outperforms the strongest visual reward baseline by 4.8 percentage points and achieves a 1.43× training speedup.

Critical strengths
  • Interpretable rewards facilitate inspection and debugging. Each reward can be traced to specific conditions, such as whether the gripper is close to an object or whether the object is inside a container. This makes it easier to identify whether an error originates from perception, task parsing, or rule definitions.

  • Continuous guidance for multistep tasks. Intermediate progress, such as approaching an object, completing a grasp, and placing it correctly, contributes to the reward. Stage monitoring also tracks execution order according to task requirements, making the approach particularly useful for long-horizon tasks.

  • Low online computational cost. The large language model is used only for offline task parsing. During training, rewards are computed through lightweight perception and explicit rules, avoiding repeated calls to large vision-language models. The paper reports that the visual reward module requires only 44 MB of model storage.

Critical weaknesses
  1. Manual effort and scalability: How much manual effort is required to extend SpecRL to new tasks, including specification review, predicate design, threshold tuning, and perception adaptation?

  2. Persistence of completed stages: Since completed stages remain recorded, how does SpecRL handle later actions that invalidate earlier achievements, such as knocking a previously placed object out of a basket?

  3. Reliability of visual rewards: How robust are the geometric predicates to perception errors, and could policies exploit these proxy conditions without achieving actual task success?

  4. Experimental validation: Can the authors provide results across multiple training seeds and real-world online RL experiments to establish the reliability and broader applicability of the reported gains?

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response-1

We sincerely thank Reviewer Amr5 for the careful reading and constructive feedback. We are encouraged that you recognized the advantages of SpecRL in terms of interpretability and online efficiency. Our point-by-point responses are detailed below.

W1: Manual effort

A: We thank the reviewer for raising this practical question. SpecRL supports task extension through reusable predicates and shared grounding configurations, with task-specific specification review performed once before RL fine-tuning. We quantify the manual setup effort below.

  1. Specification Review. Two human experts reviewed the LLM-generated specifications for all 40 LIBERO tasks, checking object references, stage ordering, and consistency with the task environment. The review times were:
MetricLIBERO-SpatialLIBERO-ObjectLIBERO-GoalLIBERO-Long
Total review time (min)5.235.925.617.45

Across all 40 tasks, review required 24.21 minutes in total, averaging approximately 36 seconds per task. For a new task, the specification review is expected to be completed within one minute. This validation is conducted once offline; the validated JSON specification is cached and reused throughout RL fine-tuning without further human intervention.

2. Predicate Design and Threshold Calibration. The seven task-level atomic propositions cover all evaluated tasks through reusable grounding functions. Across LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, threshold calibration for the privileged variant took less than 1 hour in total. For the visual variant, calibrating thresholds from visual observations at successful task completion took 2–3 hours in total across the same four suites. Once calibrated, these configurations are shared across tasks. New tasks within the scope of the LIBERO environment can reuse the existing predicates and thresholds without additional design effort.

3. Perception Adaptation. We fine-tuned separate segmentation models for simulation and real-world deployment. As reported in Appendix A, the simulation model used 11,519 frames covering 26 classes, while the real-world model used 220 demonstration frames covering 4 classes. Both models were fine-tuned for 80 epochs, requiring less than 1 hour of training in total. New tasks involving objects already supported by the segmentation model require no additional perception design. For a new object that the model does not yet support, we estimate that fine-tuning the model to recognize it would take less than 1 hour of training.

Although SpecRL-V requires a one-time offline setup, it considerably reduces RL training time. As shown in Figure 7, over 100 epochs on a server with eight A100 GPUs, SpecRL-V reduces training time by 41.78, 12.26, and 39.13 hours compared to Robo-Dopamine, TOPReward, and RoboMeter, respectively, while using less GPU memory and achieving higher performance. Even the smallest reduction of 12.26 hours exceeded the time we spent on manual design. The time savings can be even greater in longer training runs.

We have added the offline setup details to Appendix C.1.

W2: Invalidation of Completed Stages

A: We appreciate the reviewer for pointing out this case. SpecRL retains completed stages to track progress across multi-step tasks. In our LIBERO experiments, we observed no instances of later actions invalidating previously completed stages, so our current implementation does not handle such cases. Nevertheless, the specification-based design offers a natural and extensible solution to address the reviewer's concern: the task specification can be augmented with collision-avoidance safety constraints that prohibit unintended disturbances to previously placed objects or surrounding items. Any violation of these constraints can trigger an penalty term in the reward function, encouraging the policy to preserve completed stages during online fine-tuning.

We have added a discussion on how to handle this case to the revised manuscript.

AuthorsOpenReview ↗

Response-2

W3: Perception Robustness and Reward Exploitation

A: We thank the reviewer for this insightful question. First, we emphasize that the privileged variant, SpecRL-P, relies entirely on ground-truth state information and is unaffected by perception errors. Second, for the visual variant (SpecRL-V), our framework is designed to be robust against perception inaccuracies by fusing visual observations with robot proprioception rather than relying solely on a single proximity cue. For instance, the pick predicate requires the simultaneous conjunction of object–gripper proximity, proprioceptive gripper closure, and relative co-movement (Appendix C.2). This multi-cue condition prevents policies from exploiting proxy states without achieving true task success. In our visual reward comparison (Figure 8), SpecRL-V improves task success rates over TOPReward, Robo-Dopamine, and RoboMeter by 4.8, 7.2, and 15.6 percentage points, respectively, demonstrating the effectiveness of our approach.

W4: Experimental validation.

A: We thank the reviewer for this constructive suggestion. As noted in Appendix A, we follow the common practice in computationally intensive large-scale model training by using a fixed random seed [1-3], primarily due to the prohibitive computational cost. For example, our available infrastructure consists of a single server with eight NVIDIA A100 GPUs, where a single 400-epoch RL training run requires nearly four days; running multiple seeds would demand weeks of continuous computation.

In addition, following your suggestion, we have conducted real-robot online RL experiments on two manipulation tasks using a Franka robot. Following a lightweight real-world RL framework [4], we freeze the pretrained VLA and train a residual policy whose outputs are added to the VLA's proposed actions. We train this policy using either binary rewards or SpecRL-V rewards, with the same budget of 40 online interaction episodes per task. Each resulting policy is evaluated over 25 real-robot trials per task.

TaskInitial policyRL w/ BinaryRL w/ SpecRL-V
Blue bowl to pink plate72%92%100%
Blue cube to pink plate80%96%100%

Under the same interaction budget, SpecRL-V achieves 100% success on both tasks, outperforming binary rewards by 8 and 4 percentage points, respectively. These results further validate the effectiveness of our framework in real-world online RL.

We have added a subsection on real-robot online RL experiments to Section 4.5 of the revised manuscript.

[1] Li H, Zuo Y, Yu J, et al. Simplevla-rl: Scaling vla training via reinforcement learning[C]//International Conference on Learning Representations. 2026: 125846-125869.

[2] Lu G, Guo W, Zhang C, et al. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning[J]. arXiv preprint arXiv:2505.18719, 2025.

[3] Chen K, Liu Z, Zhang T, et al. πrl: Online rl fine-tuning for flow-based vision-language-action models[J]. arXiv preprint arXiv:2510.25889, 2025.

[4] Zhao W, Yan X, Wan Z, et al. BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models[J]. arXiv preprint arXiv:2609.27450, 2026.


We sincerely thank the reviewer again for the insightful comments. If you have any further questions or additional concerns, please feel free to discuss them with us.

Dear Reviewer Amr5,

Thank you again for taking the time to review our paper. Your insightful comments have helped us improve the paper.

With about a day and a half left in the discussion period, we would appreciate it if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to answer any further questions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.