acceptodds
Under review as a conference paper at ICLR 2027

VLSA-RL: Can a Small Residual Policy Enable Safety-Aware VLA Adaptation?

Abstract

How can a pretrained Vision-Language-Action (VLA) policy adapt to safety-critical manipulation tasks without updating its billions of parameters? Execution-time safety filters can prevent unsafe actions but do not teach the policy to avoid repeated interventions, while end-to-end reinforcement learning (RL) is costly. We introduce VLSA-RL, which freezes the VLA backbone and learns a 1.3M-parameter residual policy (0.037% of ) to adjust its actions before control-barrier-function (CBF) safety filtering. A safety-aware GRPO objective uses feedback from the CBF quadratic program, risk-dependent clipping, and task and safety rewards to optimize the residual policy. We further analyze how residual actions can compensate for dynamics mismatch and modulate CBF convergence behavior. On SafeLIBERO, VLSA-RL reaches a 68.0% average safe-success rate across two safety levels. Relative to the CBF-filtered AEGIS baseline, it improves safe success by 21.4 and 22.9 percentage points on Levels I and II, respectively, while reducing average episode length on both levels.

Primary area:
reinforcement learning
Keywords:
Vision-Language-Action Models, Reinforcement Learning, Safety, Control Barrier Functions, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_k86cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VLSA-RL, a method for adapting a pretrained VLA policy to safety-critical manipulation tasks without updating the VLA itself. It freezes the VLA backbone and trains a small residual policy that corrects the VLA's actions before they pass through a control barrier function (CBF) safety filter. The residual policy is optimized with a safety-aware variant of GRPO that uses feedback from the CBF filter and adjusts the update size based on trajectory-level safety, and the authors also provide an analysis of how the residual actions relate to the CBF condition. Experiments on SafeLIBERO show improved safe success rates over the CBF-filtered AEGIS baseline and other VLA policies, along with ablations of the main design choices.

Critical strengths

The motivation is clear, and the proposed method is lightweight. It avoids the cost of end-to-end RL fine-tuning by training only a small residual policy on top of a frozen VLA. The paper also establishes a connection between the residual policy and the CBF formulation, showing how residual actions can compensate for dynamics mismatch and effectively modulate the CBF convergence rate.

Critical weaknesses
  1. The novelty of the method seems somewhat incremental. The framework mainly combines existing components, including residual RL on a frozen policy, the CBF-QP safety filter from VLSA, GRPO, and the use of safety-filter corrections as a learning signal, which has also been explored in prior work cited in the paper. As a result, the contribution lies mainly in how these components are combined, and it would help if the paper more clearly highlighted what is new relative to prior work.
  2. The proposed method introduces several hyperparameters that may require tuning, including RioutR_i^{out}, λstep\lambda_{step}, cAc_A, and λQP\lambda_{QP}, among others. Although the ablation study shows the performance when each component is removed, it does not show how performance changes as these values vary. As a result, the sensitivity of the method to these hyperparameters remains unclear.
Questions for authors
  1. Since the proposed method builds on VLSA, a natural alternative is to adapt the VLA within VLSA using parameter-efficient fine-tuning methods such as LoRA, which also update only a small number of parameters. Could the authors compare the proposed residual policy with such a LoRA-based variant of VLSA, to clarify the advantages of the residual design?
  2. In Eq. (6), an unsafe success receives a higher reward (0.5) than a safe failure (0.1). This seems to prioritize task completion over safety, and may encourage the policy to accept collisions in order to complete the task. How do the authors prevent this kind of reward hacking? In addition, how are TrefT_{ref} in Eq. (7) and hcliph_{clip} in Eq. (9) defined?
  3. Several recent safe-VLA methods, including KNOWS [1] and constrained flow matching [2], are also evaluated on SafeLIBERO and appear to be directly relevant baselines. Could the authors include comparisons with these methods, which would provide a clearer assessment of VLSA-RL relative to recent approaches developed for the same safety setting?
  4. Could the authors clarify how task success is defined? Since collisions do not terminate an episode, a trajectory can still count as a task success even after colliding with an obstacle. In this case, TSR may overestimate practical task performance, while SSR appears to be the more meaningful metric for evaluating safe task completion.

[1] https://arxiv.org/abs/2606.09749
[2] https://arxiv.org/abs/2607.01378

6 comments
AuthorsOpenReview ↗

Response to Reviewer k86c (Part 1)

Thank you for your thoughtful review and for suggesting comparisons that help clarify the contribution of the residual design.

Response to W1 - Contribution

We thank the reviewer for this constructive comment. We agree that several individual components of VLSA-RL, including residual reinforcement learning, CBF-QP safety filtering, GRPO, and the use of safety-filter interventions as learning signals, have been studied in prior work. Our contribution is not to introduce these components independently, but to develop a unified safety-aware adaptation framework that integrates them specifically for pretrained VLA policies. We have further clarified the distinctions from related work in the revised manuscript.​ Specifically, our contributions lie in three aspects:

  1. Safety-aware residual adaptation of frozen VLA policies. Unlike VLSA/AEGIS, which relies on execution-time CBF-QP corrections without adapting the nominal policy, VLSA-RL trains a lightweight residual policy alongside the frozen VLA. The residual policy incorporates CBF-related state information and previous QP corrections, enabling it to learn safety-aware action refinements while retaining the original VLA and CBF-QP formulation. This also differs from existing residual RL approaches such as ResiP and PLD in its explicit integration of CBF-derived safety feedback into VLA adaptation.

  2. CBF-informed policy optimization at both action and trajectory levels. Rather than simply treating safety-filter interventions as generic penalties or reward signals, we introduce a differentiable QP correction loss to guide nominal actions toward requiring less safety intervention. Meanwhile, the minimum CBF margin over an entire trajectory determines the GRPO clipping range, explicitly coupling policy-update magnitude with trajectory-level safety risk. Together with graded task-safety rewards and fixed-scale group-relative advantages, these mechanisms establish a safety-aware optimization objective tailored to residual VLA adaptation.

  3. Theoretical characterization of residual actions in CBF-based control. We derive a sufficient condition under which residual actions compensate for dynamics mismatch in the CBF constraint. We further show that the residual contribution is algebraically equivalent to a state-dependent modulation of the CBF convergence-rate coefficient, providing a theoretical interpretation of how residual adaptation affects safety behavior.

The ablation results further support the contributions of these designs. For example, removing the CBF-QP correction loss reduces TSR and SSR from 93.0% and 74.0% to 87.5% and 69.5%, respectively. Replacing margin-dependent clipping with the best fixed clipping bound also reduces TSR and SSR to 90.0% and 72.5%. These results demonstrate the complementary contributions of action-level CBF feedback and trajectory-level risk-aware optimization.

We acknowledge that the novelty lies in the proposed integration and its resulting learning and theoretical mechanisms, rather than in any individual building block. We have revised the Introduction and Related Work to make these distinctions more explicit.

Response to W2 - Hyperparameter sensitivity

We thank the reviewer for this valuable suggestion. We acknowledge that a systematic hyperparameter sensitivity analysis has not been conducted, as it would require substantial additional training runs that we could not complete within the limited rebuttal period. Nevertheless, the hyperparameter configuration reported in the paper has been validated through our experiments and demonstrates effective task and safety performance. We recognize the importance of further investigating hyperparameter sensitivity and would be happy to conduct additional experiments and incorporate the results in a subsequent revision if there is an opportunity for further revision.

AuthorsOpenReview ↗

Response to Reviewer k86c (Part 2)

Response to Q1: Comparison with LoRA-based Adaptation

We thank the reviewer for this valuable suggestion. Following this suggestion, we have added a new Appendix I, which presents a comparison between VLSA-RL and a LoRA-based VLSA variant on SafeLIBERO Spatial I. We have also added a new Figure 12 to illustrate the full evaluation curves.

Specifically, we apply rank-9 LoRA (α=9\alpha=9) to the query, key, value, and output projections of the π0.5\pi_{0.5} action expert. The LoRA variant trains 1.41M parameters, comparable to the 1.36M trainable parameters in VLSA-RL. Both methods are trained for 300 updates and employ the same CBF-QP safety filter during execution.

An important distinction lies in how the two methods incorporate safety information. The LoRA variant retains the original VLA input interface and conditions action generation on visual observations and task instructions. Standard LoRA adaptation updates model parameters but does not inherently introduce additional input channels for explicit safety information. In contrast, VLSA-RL keeps the VLA backbone frozen and employs a separate residual policy that additionally receives a 79-D context vector containing robot states, obstacle geometry, CBF information, and previous QP corrections. This allows the residual policy to explicitly exploit safety-related information when adapting the nominal VLA actions, without modifying the original VLA architecture.

For a fair comparison, we select the best checkpoint of each method based on SSR, using TSR to break ties. The results are summarized below:

MethodBest StepTSR (%)SSR (%)ETS (steps)
VLSA-RL27593.074.0143.1
LoRA25070.550.5197.05

The newly added Figure 12 presents the full evaluation curves. VLSA-RL exhibits consistent improvement during training, whereas LoRA fails to improve SSR beyond its pre-adaptation performance. These results demonstrate that, with comparable parameter budgets and training iterations, the proposed safety-aware residual adaptation achieves substantially better task and safety performance than the evaluated LoRA-based alternative.

Overall, the advantage of the residual design is that it provides a dedicated pathway for incorporating explicit safety information into policy adaptation, rather than relying solely on modifying the parameters of the original VLA policy.

Response to Q2: Reward design and thresholds

We thank the reviewer for raising this important concern. Our tiered reward design reflects the consideration that safety without task completion is insufficient for meaningful behavior. For example, a policy that never takes action can trivially avoid collisions but is unable to accomplish the task. This issue is particularly pronounced in challenging long-horizon tasks (e.g., LONG), where successful trajectories themselves are scarce during early training, and safe successes are even rarer. Assigning partial credit to unsafe successes (0.5), rather than treating them equally with safe failures (0.1), preserves informative learning signals from task-completing trajectories and encourages the policy to develop task competence. Meanwhile, safe successes receive the highest reward (1.0), progressively encouraging collision-free task completion. Thus, our design aims to establish task competence while promoting increasingly safe behaviors, rather than sacrificing task completion for trivial safety. We also acknowledge that alternative reward designs may offer different trade-offs between task completion and safety, which would be valuable to investigate in future work.

We apologize for the omission of these parameter values in the original manuscript. We now report them in Training Details (Section 4.1). Specifically, we set TrefT_{\mathrm{ref}} to 150 steps for the Spatial, Goal, and Object suites, and to 350 steps for the Long suite. We set hclip=0.02h_{\mathrm{clip}}=0.02 for all suites and both safety levels.

AuthorsOpenReview ↗

Response to Reviewer k86c (Part 3)

Response to Q3: KNOWS and constrained flow matching

We thank the reviewer for this valuable suggestion. Following this suggestion, we have added new comparisons with KNOWS and constrained flow matching to Tables 1 and 2, reporting their results on SafeLIBERO Level I and Level II, respectively. We have also cited both works in the revised manuscript. Their CAR results have also been added to the new Tables 9 and 10 in Appendix F. The suite-averaged results are summarized below.

MethodLevel I TSR (%)Level I SSR (%)Level I ETS (steps)Level II TSR (%)Level II SSR (%)Level II ETS (steps)
VLSA-RL80.063.1225.1782.872.8217.73
KNOWS67.559.9NR71.153.8NR
Constrained FM84.6NR275.8778.6NR324.07

KNOWS does not report ETS. Constrained flow matching reports TSR, CAR, and ETS, but does not report SSR. We therefore mark its SSR as NR, since the separately reported task success and collision avoidance rates do not determine the joint rate of safe task completion.

We note that the source code of these methods is not publicly available, making direct reproduction under our experimental settings difficult. Furthermore, constrained flow matching uses simulator-provided obstacle positions, and its episode step limits are not specified. Some of its reported task-level ETS values also exceed our evaluation limits. Therefore, we retain the results reported in the original papers under their respective evaluation protocols and present them as cross-paper references rather than strictly controlled comparisons.

Response to Q4: Task success

We thank the reviewer for pointing out this important clarification. We agree that TSR alone may overestimate practical task performance because collisions do not terminate an episode. We have therefore clarified the task-success definition in Section 4.1. Specifically, an episode counts toward TSR if it satisfies the benchmark's task-success condition within the specified episode step limit, which is 300 steps for the Spatial, Goal, and Object suites and 550 steps for the Long suite. A collision marks the episode as unsafe but does not terminate the rollout. Thus, an episode that completes the task after a collision still counts toward TSR, whereas SSR only counts episodes that complete the task without any detected collision.

Summary of Revisions

  • Added a LoRA-based comparison in new Appendix I and the full evaluation curves in new Figure 12.
  • Added comparisons with KNOWS and constrained flow matching to Tables 1 and 2, and added their CAR results to the new Tables 9 and 10 in Appendix F.
  • Reported the values of TrefT_{\mathrm{ref}} and hcliph_{\mathrm{clip}} in Training Details (Section 4.1).
  • Clarified the task-success definition and episode step limits in Section 4.1.

Closing remarks
We would be happy to address any further questions you may have and hope that our revisions and responses will lead to a more favorable assessment. Thank you again for your time and constructive feedback.

Reviewer_jFEVOpenReview ↗
Rating
4(+3)
Confidence
4
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper introduces a small residual policy that plugs into large pretrained VLA backbones and ensures the VLA policy behaves safely. The method uses a Control barrier function to filter policy actions so that it is always modified to produce a safe action. The authors introduce a safety aware GRPO update and a differentiable CBF projection loss that ensures the policy learns to minimize correction and the residual policy gets updated without modifying backbone weights. Experiments with Franka Panda Arm having 7-DOF shows considerable improvement in safety, task success and execution steps compared to chosen baselines.

Critical strengths

This paper combines several published techniques into one coherent formulation that provides considerable improvements over existing techniques. The major strength is the efficiency of the overall policy. By creating an adapter policy that only modifies the final action stream without changing any of the backbone weights allows it to be quite tiny and negligible in terms of its execution time. Additionally, using a CBF provides guarantees on the actions produced under standard model dynamics mismatch.

Critical weaknesses

I have concerns about some lack of details in the paper, and about experiments.

  1. The CBF usage is not really clear to me. A standard CBF requires differentiable dynamics, but I did not see any learned or ready-available dynamics being used in the formulation. Appendix C.1 has thee CBF expressed in detail but it is still not clear to me. Secondly, the CBF uses sts_t, which I assume is an internal simulator state having the actual physical state of the simulation, while none of the policy networks, VLA backbones or any other component use sts_t. This might be a notation issue and could be corrected, but details of what sts_t is, if other baselines use it are not provided.

  2. If sts_t and dynamics are available, this reduces to a standard safe RL problem then. Why cant techniques like SauteRL, RCPO etc be used (https://github.com/PKU-Alignment/omnisafe)? I am sure they could use a frozen VLA backbone, similar residual policy and the safety function to learn safe behaviour. A CBF safety work like RAMPS (https://openreview.net/forum?id=2bbqHOWFTU) could also be appropriate here.

  3. A critical weakness of this paper is a well-defined CBF function, which might not be available for most problems. This limits broader generalizability of this work. It shows up in the limited experiments and domains used in the paper. Safety-gymnasium https://safety-gymnasium.readthedocs.io/en/latest/ has a lot of benchmarks which are vision specific, have partially observable environments which stress-test actual safety-critical tasks. I would like to see tests on these.

  4. The CBF-specific loss is also not clear to me. If we try to pull the CBF action closer to the policy action rather than the opposite, we are minimizing correction rather than safety. I would expect to pull the nominal action closer to the CBF action and leave the CBF untouched.

Questions for authors
  1. What is sts_t, is it used in the policy network or any state? If not, how is the CBF differentiable? Could you specify the dynamics function used?

  2. The current paper very specific to the franka arm used. How difficult would it be to extend to something like safe-navigation?

11 comments
AuthorsOpenReview ↗

Response to Reviewer jFEV (Part 1)

We thank the reviewer for the careful assessment and the constructive suggestions. We appreciate the opportunity to clarify our findings and provide additional analyses.

Response to W1 & Q1: Nominal dynamics and the CBF state

We apologize that the original manuscript did not explain these details clearly.

Regarding the dynamics model, we use the analytical end-effector kinematics adopted in VLSA. Specifically, Appendix C.2 now explicitly gives

\dot\varphi=0.2a_{4:6},$$ for position and orientation, respectively. The virtual unit vector $p_s$, which determines the supporting hyperplane, evolves according to $$\dot p_s=(I-p_sp_s^\top)\nu.$$ Regarding the state $s_t$, it denotes the augmented CBF state, defined as $s_t=(p_{\mathrm{ee}},\varphi,p_s)$, comprising the end-effector pose and the virtual tangent-plane variable. These state components are also provided to the residual policy through the 79-dimensional context vector $z_t$. Specifically, as shown in Appendix D, Table 4, entries 1–3 encode the end-effector position, entries 4–7 encode its orientation as a quaternion, and entries 48–50 correspond to $p_s$. The context additionally includes the ellipsoid parameters, the current barrier value $h(s_t)$, and the previous QP correction. Regarding CBF differentiability, the safety layer directly uses $s_t$ and computes the barrier derivative through the chain rule: $$\dot h(s,a)=\nabla_s h(s)^\top\dot s.$$ The closed-form barrier function in Appendix C.1 is differentiable with respect to the modeled state variables when the obstacle geometry is fixed, while $\dot s$ is determined by the analytical dynamics above. This computation is independent of the policy's input representation and yields an affine CBF constraint on the control action (Eq. (3)). Furthermore, for a fixed state, the QP projection is differentiable almost everywhere with respect to the nominal action, as established in Appendix C.2. Finally, AEGIS uses the same state information for its execution-time CBF-QP. The frozen VLA backbone in our method and the standalone VLA baselines retain their original policy inputs. **Response to W2: Safe RL and RAMPS** We thank the reviewer for suggesting SauteRL, RCPO, and RAMPS. We agree that these methods are relevant and could, in principle, be combined with a frozen VLA backbone and a residual policy. However, our work addresses a different research question: how to adapt a pretrained VLA policy to safety-critical manipulation tasks by learning directly from safety-filter interventions, without updating the VLA backbone. SauteRL and RCPO primarily address constrained policy optimization through safety-budget augmentation and cost-based optimization, respectively. In contrast, VLSA-RL explicitly exploits CBF-QP feedback during policy learning. Specifically, (1) the previous QP action correction is incorporated into the residual policy's input, (2) a differentiable CBF-QP loss encourages the nominal actions to require smaller safety corrections, and (3) the trajectory-level CBF margin determines the clipping range in our safety-aware GRPO objective. These mechanisms allow the residual policy to exploit action-level safety information beyond trajectory-level task and safety outcomes. RAMPS focuses on robust multi-step predictive shielding using learned dynamics, aiming to improve the safety of action execution under model uncertainty. In comparison, our primary contribution lies in safety-aware adaptation of a frozen, pretrained VLA policy. These two directions are complementary: a more advanced predictive safety shield could potentially be integrated into our framework. We do not claim that existing safe RL algorithms are inapplicable or inherently inferior. Rather, our contribution is a VLA-specific residual adaptation framework that integrates CBF-derived safety feedback into policy optimization while retaining execution-time safety filtering. **We have added citations to SauteRL, RCPO, and RAMPS and clarified how VLSA-RL differs from these methods in Section 2 and Appendix A of the revised manuscript.**
AuthorsOpenReview ↗

Response to Reviewer jFEV (Part 2)

Response to W3: CBF scope and evaluation

We thank the reviewer for this valuable comment. Although our CBF formulation may appear mathematically involved, its underlying geometric interpretation is relatively simple. Specifically, we approximate the robot end effector and each obstacle using minimum-volume enclosing ellipsoids, and formulate collision avoidance as a separation problem between two ellipsoids. Thus, rather than manually designing a task-specific CBF for each obstacle shape, our method provides a unified geometric construction applicable to a broad range of bounded obstacles. The CBF measures their geometric separation using a rotating supporting hyperplane, whose interpretation is illustrated in the newly added schematic figure in Appendix C.1. We acknowledge that ellipsoidal approximations may introduce conservatism for highly non-convex obstacles and that the method relies on sufficiently accurate geometric perception.

Regarding Safety-Gymnasium, we appreciate the suggestion and have cited this valuable benchmark in the manuscript. While it provides diverse vision-based and partially observable environments, our work specifically focuses on safety-aware adaptation of pretrained, language-conditioned VLA policies for robotic manipulation. Safety-Gymnasium does not directly provide a pretrained VLA policy and action interface compatible with our π0.5-based setup. Adapting our framework would therefore require additional VLA integration and task-specific CBF construction, introducing factors beyond the scope of this study. We choose SafeLIBERO because it provides a directly compatible evaluation setting for isolating the effectiveness of our proposed safety-aware residual adaptation. Extending our framework to Safety-Gymnasium and other partially observable environments is an important direction for future work.

Response to W4: Clarification of the CBF-QP Constraint Loss

We thank the reviewer for raising this important point. We would like to clarify that our CBF-QP loss already encourages the nominal policy action to approach the CBF-projected safe action, rather than the reverse.

Specifically, the CBF constraints and the QP projection operator are not trainable, and only the parameters of the residual policy are optimized. For a fixed state ss, let C(s)={a~:ws⊤a~≥bs}\mathcal{C}(s)=\{\tilde a : w_s^\top\tilde a\geq b_s\} denote the CBF-feasible action set. As defined in Eq. (34), the QP output is the Euclidean projection of the nominal action onto this set:

a~⋆=ΠC(s)(a~nom).\tilde a^\star=\Pi_{\mathcal{C}(s)}(\tilde a^{\mathrm{nom}}).

Since the joint correction in Eq. (35) is parallel to wsw_s, the per-step loss in Eq. (12) is

ℓQP=c(s)dist⁡2(a~nom,C(s)),c(s)=∣ws,1:6∣22dQP∣ws∣22.\ell_{\mathrm{QP}}=c(s)\operatorname{dist}^2(\tilde a^{\mathrm{nom}},\mathcal{C}(s)),\qquad c(s)=\frac{\\|w_{s,1:6}\\|_2^2}{d_{\mathrm{QP}}\\|w_s\\|_2^2}.

For fixed ss and nominal auxiliary control, its gradient with respect to the nominal robot-motion components is

∇a~1:6nomℓQP=2c(s)(a~1:6nom−a~1:6⋆).\nabla_{\tilde a^{\mathrm{nom}}_{1:6}}\ell_{\mathrm{QP}}=2c(s)(\tilde a^{\mathrm{nom}}_{1:6}-\tilde a^\star_{1:6}).

Therefore, gradient descent explicitly moves the nominal action toward its CBF projection, which is precisely the optimization direction suggested by the reviewer. Importantly, differentiating through the projection does not modify the CBF constraints or optimize the safety filter itself.

The loss encourages nominal actions to satisfy the modeled CBF constraints and thereby require smaller execution-time corrections. The CBF-QP layer remains responsible for enforcing these constraints at execution time under the stated modeling assumptions.

AuthorsOpenReview ↗

Response to Reviewer jFEV (Part 3)

Response to Q2: Extension to Safe Navigation

Thank you for this question. Although our current implementation is evaluated on a Franka Panda arm, the proposed framework is not inherently restricted to robotic manipulation. Its extension to safe navigation, including vision-and-language navigation (VLN), mainly requires adapting the system dynamics, CBF formulation, and policy state representation to the target platform.

For example, in mobile robot navigation, the robot footprint and obstacles can similarly be approximated using ellipsoids, allowing the same geometric separation-based CBF construction to be applied. The control-affine dynamics can be reformulated according to the robot's motion model (e.g., a unicycle model with linear and angular velocity inputs), from which the corresponding CBF-QP can be derived. Furthermore, a pretrained VLN policy conditioned on visual observations and language instructions can serve as the nominal navigation policy, while a lightweight residual policy learns safety-aware action corrections through CBF-QP feedback. Importantly, the VLA/VLN backbone can remain frozen, and the overall residual adaptation framework remains unchanged. Therefore, extending VLSA-RL to safe navigation and VLN is technically feasible, although platform-specific dynamics and safety constraints need to be established.

Summary of Revisions

  • Added the explicit nominal end-effector dynamics and clarified the augmented CBF state and its use in Appendix C.2.
  • Added a new schematic figure in Appendix C.1 to illustrate the geometric interpretation of the CBF.
  • Added citations to SauteRL, RCPO, and RAMPS, and clarified the distinctions from VLSA-RL in Section 2 and Appendix A.

Closing remarks
We would be happy to address any further questions you may have and hope that our revisions and responses will lead to a more favorable assessment. Thank you again for your time and constructive feedback.

Reviewer_2heXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the use of a residual policy that corrects the frozen VLA's actions before a CBF based safety filter. The residual is trained with a variant of GRPO that adds a tiered safe/unsafe x success/failure reward, fixed-scale advantages, a clip range that depends on the trajectory's minimum CBF margin and a curriculum. The paper provides a sufficient condition for the residual to compensate for dynamics mismatch and interprets the residual as a state-dependent quantity of the CBF rate. Experiments show that VLSA-RL improves safe success-rate (SSR) over AEGIS signifncalty.

Critical strengths
  1. Using a residual policy to finetune the VLA by limiting the number of trainable parameters is the right approach to ensure safety. Feeding the previous QP correction back into the residual's input is a sensible choice.

  2. Experiments do show large, consistent gains over AEGIS. SSR improves in 8 of 8 suite × level settings and TSR in 7 of 8, and episodes are shorter.

Critical weaknesses
  1. Important ablations are missing: a) a plain residual-RL baseline (binary SSR reward, vanilla GRPO, CBF at execution); b) evaluation without the CBF-QP at test time; c) the QP-intervention rate before and after training. This would test the central claim that the policy learns to need fewer interventions.

  2. It is unclear whether the evaluation measures adaptation or fitting to the test episodes. Each residual is trained on "50 training episodes per task" and evaluated on "50 evaluation episodes" per task. The paper never states that these sets are disjoint, and under LIBERO's fixed-initial-state protocol they would be identical.

  3. The margin-dependent clip range is the only completely novel ingredient that the paper proposes and Table 3 cannot distinguish it from a fixed ε. all other ideas already exist in literature.

Questions for authors
  1. Are the training and evaluation initial configurations identical? Please report results on held-out configurations, using the final checkpoint, with at least 3 seeds and confidence intervals.
  2. Please add three comparisons: (i) a plain residual-RL baseline, (ii) the method without the CBF-QP at test time, and (iii) the QP-intervention rate and ‖δa^QP‖ over training.
  3. What does Theorem 1 imply for the executed (post-projection) action? Can you show mismatch compensation empirically, for example with added latency or a changed payload?
  4. Please report the collision-avoidance rate for all methods, and break collisions down by source (end-effector, other links, grasped object).
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2heX

We thank the reviewer for the careful assessment and the constructive suggestions. We appreciate the opportunity to clarify our findings and provide additional analyses.

Response to W1 & Q2: Additional ablations

We thank the reviewer for the questions regarding the individual contributions of the proposed components. We provide the following additional results and clarifications.

(1) Standard GRPO. We have added a comparison with a residual baseline trained using standard GRPO to Table 3. The residual baseline uses a binary safe-success reward, while retaining the same CBF-QP filter at execution.

Spatial I variantTSR (%)SSR (%)ETS (steps)
Standard GRPO83.570.0150.9
VLSA-RL93.074.0143.1

The improvements under the same execution-time filter support the effectiveness of safety-aware GRPO for residual policy training.

(2) Removing the filter (new Appendix H and Table 12). To further examine the contribution of residual learning independently of the execution-time safety filter, we evaluated both policies with the execution-time filter disabled.

Policy (filter off)Spatial I TSR (%)Spatial I SSR (%)Spatial II TSR (%)Spatial II SSR (%)
π0.5\pi_{0.5}65.014.553.513.0
VLSA-RL77.518.580.028.0
Gain (percentage points)+12.5+4.0+26.5+15.0

In computing SSR, we classify any episode with a detected collision as unsafe, regardless of collision severity. Although the gains in SSR are smaller, the newly added Figure 11 shows several cases in which contact under the residual policy caused visibly less disturbance to the obstacles. These milder contacts may, in turn, have allowed more episodes to reach task completion after a collision, partly explaining the larger gains in TSR.

(3) QP interventions (new Figure 6). We also report the QP intervention statistics to characterize how the safety filter is used during training. An intervention is counted when the six-dimensional robot-action correction norm exceeds 10−610^{-6}.

MetricFirst 25 updatesLast 25 updates
Mean intervention steps per episode65.3652.91
Mean cumulative correction norm per episode25.1019.82
Mean episode length (steps)210.49176.31
Per-step intervention rate (%)31.0530.01
Mean correction norm per executed step0.11920.1124

As episodes became shorter, both per-step measures also declined modestly.

Response to W2 & Q1: Unseen initial configurations

We thank the reviewer for raising the question of generalization to unseen initial configurations. The original protocol used the same fixed pool of 50 initial configurations per task for RL training and evaluation, similar to the within-task LIBERO setup in the public implementation of SimpleVLA-RL. The original manuscript also reported a separate generalization test, which provided initial evidence of performance beyond the training conditions. To complement that test, we added a new evaluation on unseen initial configurations within the same Spatial I tasks.

For this evaluation, we generated 50 new configurations for each of the four Spatial I tasks, yielding 200 configurations in total. We translated movable objects by 3-8 mm in the horizontal plane, perturbed their yaw angles within ±3∘\pm 3^\circ, and perturbed the seven robot joint angles within ±1∘\pm 1^\circ. To preserve relative poses, objects in support relationships were moved together. We then checked each configuration for physical validity independently of policy outcomes.

We evaluated VLSA-RL and AEGIS on this shared set of 200 configurations using three matched evaluation seeds. The results are reported in the new Appendix G, Table 11, with means and 95% Student's t confidence intervals across these seeds.

MethodTSR (%)SSR (%)ETS (steps)
AEGIS67.17 [59.68, 74.65]45.83 [37.56, 54.10]201.25 [185.70, 216.80]
VLSA-RL90.00 [86.71, 93.29]72.17 [67.15, 77.19]148.12 [142.83, 153.40]

The resulting 26.3-percentage-point gap indicates that VLSA-RL's advantage in safe task completion over AEGIS also extends to unseen initial configurations within the tested perturbation ranges.

AuthorsOpenReview ↗

Response to Reviewer 2heX (Part 2)

Response to W3: Adaptive clipping and novelty

We thank the reviewer for this constructive comment. We agree that several individual components of VLSA-RL build on established techniques. However, our contribution lies not only in combining these components, but also in how CBF-derived safety information is integrated into residual policy optimization at both action and trajectory levels.

Regarding the adaptive clipping mechanism, Table 3 compares different clipping rules on Spatial I while keeping the other components fixed. Our margin-dependent clipping dynamically adjusts ϵ\epsilon within [0.1,0.2][0.1, 0.2]. Importantly, in our experiments, the mean clipping value produced by this strategy is approximately 0.155, which is very close to the fixed value of 0.15. Nevertheless, the adaptive strategy outperforms all three fixed clipping settings, including both endpoints of its clipping range, achieving higher TSR (93.0% vs. 90.0% and 85.0%) and SSR (74.0% vs. 72.5% and 69.5%) than fixed ϵ=0.1\epsilon=0.1 and ϵ=0.2\epsilon=0.2, respectively. It also requires fewer execution steps. These results suggest that the improvement comes from assigning clipping ranges according to individual trajectories' safety margins, rather than simply choosing a particular fixed clipping value. Specifically, trajectories with lower CBF margins receive wider clipping ranges, while safer trajectories receive tighter ranges.

Spatial I clipping ruleTSR (%)SSR (%)ETS (steps)
Margin-dependent ϵ\epsilon93.074.0143.1
Fixed ϵ\epsilon = 0.1090.072.5145.6
Fixed ϵ\epsilon = 0.1589.072.0151.9
Fixed ϵ\epsilon = 0.2085.069.5158.7

More broadly, VLSA-RL exploits the same CBF-QP filter to guide residual learning at two complementary levels: (1) its action corrections provide an action-level QP loss that encourages nominal actions to approach the CBF-feasible set; and (2) its trajectory-level CBF margins determine the clipping ranges in safety-aware GRPO. Furthermore, Theorem 1 characterizes the conditions under which residual actions can compensate for dynamics mismatch in the CBF constraint, providing a theoretical interpretation of the role of residual adaptation in safety-critical control.

Therefore, we believe that the contribution of VLSA-RL is not limited to the adaptive clipping mechanism itself, but lies in the integration of CBF-informed action-level and trajectory-level learning signals, together with the theoretical characterization of residual adaptation for frozen VLA policies.

Response to Q3: Dynamics mismatch and the executed action

We thank the reviewer for raising this question about the relationship between the residual action, dynamics mismatch, and the executed action. The CBF-QP evaluates actions using a nominal dynamics model, while the residual policy learns from the task and safety outcomes of executing those actions in the environment. It can therefore learn corrections that counteract the effects of model mismatch.

Theorem 1 provides a formal characterization of this effect: if the residual action adds enough CBF margin to cover the loss caused by mismatch, the action satisfies the CBF condition under the actual dynamics. For h(s)>0h(s) > 0, this added margin can be written as Δκres(s)h(s)\Delta\kappa_{\mathrm{res}}(s)h(s), so the residual action has the same effect in the CBF inequality as a state-dependent adjustment to κ\kappa. We note that tuning κ\kappa is an established technique in CBF-based control to balance constraint feasibility and control performance (e.g., Zeng et al., 2021).

Whether this margin carries over to execution depends on the projection. If the nominal joint action already satisfies the CBF constraint, the QP leaves it unchanged and preserves the margin supplied by the residual. If the QP projects the action onto the constraint boundary, the nominal slack is zero. Thus, at execution, the relevant question is whether the nominal margin at the projected action is large enough to offset the adverse effect of mismatch at that action.

Our experiments already use an approximate model. Specifically, it maps translation and rotation commands to end-effector position and orientation rates with a constant gain of 0.2, thereby simplifying the arm's low-level response. With the same nominal model and CBF-QP filter as AEGIS, VLSA-RL improves average SSR by 21.4 and 22.9 percentage points on SafeLIBERO Levels I and II, respectively. These results provide preliminary evidence that residual adaptation is effective with approximate dynamics. We agree that further validation would be valuable and could be pursued through controlled latency and payload experiments to test mismatch compensation directly.

AuthorsOpenReview ↗

Response to Reviewer 2heX (Part 3)

Response to Q4: Collision avoidance and contact sources

We thank the reviewer for the question regarding collision avoidance and the sources of contact. The newly added Tables 9–10 in Appendix F report the collision avoidance rate (CAR) for every method in the main comparison, broken down by task suite and safety level. CAR is the fraction of episodes with no detected collision, including collision-free task failures.

Table 9. Collision avoidance rates on SafeLIBERO Level I (%)

MethodSpatialGoalObjectLongAvg.
π0.5\pi_{0.5}14.54.014.514.511.9
OpenVLA-OFT12.522.00.00.58.8
GR00T-N1.713.00.00.06.04.8
UniVLA15.513.57.54.510.3
OptimusVLA12.01.518.015.511.8
SafeDojo69.588.557.575.572.8
KNOWS67.590.093.554.576.4
Constrained FM77.594.082.582.084.0
AEGIS57.088.534.035.553.8
VLSA-RL76.098.573.047.573.8

Table 10. Collision avoidance rates on SafeLIBERO Level II (%)

MethodSpatialGoalObjectLongAvg.
π0.5\pi_{0.5}13.536.021.518.522.4
OpenVLA-OFT4.017.023.011.513.9
GR00T-N1.710.05.543.510.017.3
UniVLA12.014.535.529.522.9
OptimusVLA12.523.522.012.517.6
SafeDojo79.566.583.066.073.8
KNOWS73.059.082.560.568.8
Constrained FM75.582.585.583.081.6
AEGIS59.562.072.065.064.6
VLSA-RL85.596.084.570.084.0

In our SafeLIBERO evaluations, we flag a collision when the selected obstacle's L1 displacement from its initial position exceeds 1 mm. The original experiments recorded collisions at the episode level. Distinguishing contacts involving the end effector, other arm links, or a grasped object would require the contact geometry IDs and the grasp state at the time of contact. In the rollouts we inspected, most visible collisions appeared to involve the arm's penultimate link when the obstacle was correctly identified. We observed this pattern for both VLSA-RL and AEGIS. We acknowledge that a quantitative breakdown by contact source would require additional logging; we would be happy to collect the relevant contact and grasp information in follow-up evaluations.

Summary of Revisions

  • Added the Standard GRPO comparison to Table 3.
  • Added the evaluation without the execution-time safety filter in new Appendix H and Table 12, together with qualitative examples of reduced obstacle disturbance in new Figure 11.
  • Added the QP intervention analysis in new Figure 6.
  • Added the evaluation on unseen initial configurations in new Appendix G and Table 11.
  • Added CAR results for the compared methods in Appendix F, new Tables 9 and 10.

Closing remarks

We would be happy to address any further questions you may have and hope that our revisions and responses will lead to a more favorable assessment. Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.