acceptodds
Under review as a conference paper at ICLR 2027

Edit Only Where It Fails: Boundary-Guided Editing via Latent Transformations for Vision-Language-Action Models

Abstract

Vision-language-action models provide strong priors for robot manipulation, but their performance can degrade under deployment-specific differences in object geometry, viewpoint, contact conditions, and robot embodiment. Existing adaptation methods often modify the policy globally, even when deployment failures are concentrated in a limited set of states. This creates a tension between adaptation and preservation: sufficiently strong updates may correct new failures but also disturb useful pretrained behaviors. We present BELT-VLA, a boundary-guided policy editing framework for online reinforcement learning of vision-language-action models. BELT-VLA identifies failure-prone regions through human interventions and applies state-conditioned orthogonal transformations to the action-flow representation only when an edit is required. The edit policy is optimized with an objective that balances exploration against intervention risk, while the orthogonal editing mechanism limits unnecessary policy deviation. We evaluate the method on four real-world manipulation tasks involving precise insertion, assembly, packing, and liquid transfer. The proposed approach increases the average success rate from 33.5% to 82.0%, while reducing human intervention and limiting policy changes outside failure-critical states. Comparisons with parameter-efficient fine-tuning and unconstrained adaptation show that boundary-guided orthogonal editing provides a favorable balance between online policy improvement and prior preservation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action model, reinforcement Learning, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YKMZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BELT-VLA, an online RL framework for adapting pretrained VLA models. It uses human interventions to identify failure-prone states and applies orthogonal transformations to edit latent action features. The method combines RND novelty and failure risk for optimizaiton, showing some improvments on four real-world manipulation tasks.

Critical strengths

I think the motivation is reasonable, and localized policy editing is an intresting direction. The paper also provides real-world robot experiemnts, and the reported results are promising.

Critical weaknesses
  1. Limited novelty and questionable methodology. The method mainly combines existing techniques, including PPO, RND, Failure Critic and orthogonal transformations. I don't see much methodological novelty here. More importantly, why can optimizing novelty and intervention risk improve task success without directly using task rewards? This is quite confusing and needs more explaination and evidance.

  2. The experiments are not very convincing. Although the success rate improves from 33.5% to 82%, the improvement over HIL-SERL is only 4.5 points. Considering the additional complexity, I don't think this improvment is very impressive. Also, only four tasks are evaluated, without sufficient statistical analaysis or more comprehensive comparisions.

  3. The preservation claim is not well supported. The authors mainly use latent and action deviations to measure policy preservation. However, smaller deviations do not neccessarily mean the original VLA capabilities are preserved. I think the authors should directly evaluate the adapted policy on other tasks. The reliability and generalizaiton of the Failure Critic also need further evaluation.

Questions for authors

No more further question. Please see the weakness part.

1 comment

We thank for your careful evaluation and constructive feedback. We respond to the main concerns below.

Q1: Limited Novelty and Questionable Methodology

Our key contribution is an intervention-guided editing framework that couples where to edit (risk-conditioned regularization) with how to edit (orthogonal latent transformations). The same continuous risk estimate ρt\rho_t guides both reward shaping and edit regularization:

rt=αrtcur−βρt,r_t=\alpha r_t^{cur}-\beta\rho_t,

=\mathbb E\left[(1-\rho_t)\|v_t\|_2^2\right].$$ Here, $r_t^{cur}$ is the RND novelty reward, $\rho_t\in[0,1]$ is the intervention risk predicted by the Failure Critic, and $v_t$ is the latent editing vector. The risk penalty discourages intervention-prone states, while edit regularization suppresses unnecessary changes in low-risk states. Unlike LAE [1] and ZPRL [2], BELT-VLA jointly uses intervention-derived risk to guide online optimization and state-conditioned orthogonal editing. Our reward design was motivated by an observation when reproducing real-world RL methods: human interventions led to successful episodes with a terminal reward of ‘1’. Thus, terminal success rewards may not reflect the reliability of autonomous execution. The pretrained VLA provides task priors, while RND encourages exploration and intervention risk discourages unreliable actions. Ablations support both components: removing RND novelty or the risk penalty reduces success rates from 82.0% to 70.5% and 66.0%, respectively. Editing ablations further validate this design: BELT-VLA achieves 82.0% success, compared with 72.0% without boundary regularization and 68.5% with unconstrained editing. We also provide more detailed reward comparisons in the new **Effects of Reward Design** subsection. **Q2: The Experiments Are Not Very Convincing** We acknowledge the modest average improvement over HIL-SERL. However, on the most challenging task (screwdriver assembly), BELT-VLA improves success from **64.0% to 72.0%** (+8.0 percentage points). Moreover, BELT-VLA updates fewer than 1M parameters while keeping the pretrained VLA frozen, offering lightweight adaptation under the same interaction budget. To provide additional statistical evidence, we report 95% confidence intervals. Across 200 test trials, BELT-VLA achieves 82.0% success (95% CI: 76.1–86.7%), compared with 77.5% (95% CI: 71.2–82.7%) for HIL-SERL. The 4.5-point improvement has an approximate 95% CI of −3.4 to 12.3 percentage points. BELT-VLA outperforms HIL-SERL on three tasks and matches it on the fourth. We appreciate the reviewer's suggestion for broader evaluation. Although limited to four tasks, they cover diverse manipulation skills, such as object transport, precise alignment, and small-object grasping, providing an initial evaluation across different manipulation challenges. We are working to add further baselines and simulation benchmarks to assess robustness and generalizability. **Q3: The Preservation Claim Is Not Well Supported** We agree that smaller action deviations do not directly demonstrate cross-task capability preservation. BELT-VLA primarily targets **within-task behavioral preservation**, minimizing changes in reliable states while correcting failures. We keep the pretrained VLA frozen and use risk-conditioned regularization. Experiments show much smaller action deviations in reliable states (0.07) than near failure boundaries (0.29). Removing boundary regularization reduces the success rate from 82.0% to 72.0%. For the Failure Critic, we further evaluate its reliability on held-out trajectories. Using intervention labels with $K=3$, we obtain an average **AUROC of 0.82** and **AP of 0.40**. These results support reliable intervention-risk discrimination on the evaluated tasks. We acknowledge that cross-task generalization remains insufficiently evaluated, particularly under limited intervention data. Improving the robustness of failure-risk estimation is an important direction for future work. **References** [1] S. Das et al., “Latent Activation Editing: Inference-Time Refinement of Learned Policies for Safer Multirobot Navigation,” arXiv:2509.20623, 2025. https://arxiv.org/abs/2509.20623 [2] D. Yu et al., “Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning,” arXiv:2605.19919, 2026. https://arxiv.org/abs/2605.19919
Reviewer_5BS1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies deployment-time adaptation of pre-trained vision-language-action (VLA) policies, where failures are typically concentrated in narrow, contact-sensitive phases (e.g., alignment, insertion) while the rest of the trajectory remains reliable. It proposes BELT-VLA, a boundary-guided policy editing framework with three components: (i) a Failure Critic trained on human-takeover timing that maps rollout states to a continuous intervention-risk score, defining a failure boundary; (ii) a frozen VLA (π0.5) augmented with a state-conditioned edit policy that applies Cayley-parameterized orthogonal transformations to the latent action-flow features; and (iii) online PPO optimization driven by RND novelty and intervention risk, with a boundary-conditioned regularizer that suppresses edits in low-risk states. On four real-world ALOHA bimanual tasks, BELT-VLA improves the average success rate from 33.5% (offline initialization) to 82.0%, while reducing the human intervention rate from 60% to 8%. A locality analysis shows edits concentrate near intervention-defined failure boundaries.

Critical strengths

A coherent "where to edit / how to edit" decomposition that is directly verified. The combination of an intervention-supervised Failure Critic (where), state-conditioned orthogonal editing of action-flow latents (how), and risk-conditioned edit regularization forms a principled answer to the stability–plasticity tension in VLA deployment adaptation.

Credible real-world evaluation of the practical claim. Four contact-rich bimanual tasks, 50 autonomous trials per task, matched interaction budgets and intervention protocols against HIL-SERL and RL-Token, and human intervention rate reported alongside success rate. And The real-world video demonstrations show promising results.

Critical weaknesses

The online objective contains no task-success signal, and this design choice is load-bearing yet untested. The reward is rt=αrtcur−βρtr_t = \alpha r^{cur}_t - \beta \rho_t (novelty minus intervention risk). The 33.5% → 82.0% improvement therefore rests on the implicit assumption that takeover timing is a faithful proxy for task failure and that novelty correlates with task progress. Silent failures that do not trigger takeover would leave the editor unguided, and no variant with even a sparse terminal success reward or a learned success classifier (as in HIL-SERL) on the same editing interface is evaluated. This makes it hard to attribute the gains to the editing mechanism versus the particular reward design, and is my main concern about generality.

Evaluation breadth and sensitivity are limited relative to the strength of the claims. A single embodiment and four tasks with no simulation benchmark. RL-Token is a reimplementation and HIL-SERL an adaptation, and the most closely related intervention-supervised post-training method (DexHiL) and inference-time steering (VLA-Pilot) are discussed but not benchmarked. There is no sensitivity analysis for the pre-intervention window KK, reward coefficients α/β\alpha/\beta, λedit\lambda_{edit}, or operator variability in takeover timing — even though the Failure Critic's estimation quality determines where edits are permitted.

The approach still relies on human intervention, limiting its applicability and scalability.

Questions for authors

Did you evaluate adding a task-success term (terminal reward or learned success classifier) to the online objective? If it helps, why omit it; if it hurts, please explain the mechanism. How does BELT-VLA behave under failure modes that do not trigger human takeover?

How sensitive are the results to KK, α/β\alpha/\beta, and λedit\lambda_{edit}? How consistent is the Failure Critic across operators with different takeover thresholds, e.g., a more conservative operator who intervenes earlier?

Can you add DexHiL-style intervention-aware post-training and VLA-Pilot-style inference-time steering under the same interaction budget, or clarify why they are not comparable? Note that DexHiL uses intervention segments as corrective supervision, which is arguably the most direct alternative use of the same human data.

2 comments

We thank the reviewer for the positive feedback and constructive suggestions. We address each concern below.

Q1: Task Success Rewards

In human-in-the-loop RL, manual recovery often leads to successful episodes despite unreliable autonomous execution. Thus, episode-level success rewards may not reflect autonomous policy reliability. Following the suggestion, we add a sparse task success reward based on autonomous episode outcomes:

=r_t+r_t^{\mathrm{task}}.$$ BELT-VLA + Task Reward achieves 83.5% average success rate, a modest 1.5-point gain over the original 82.0%. We thank the reviewer for this valuable suggestion, as task rewards may offer greater benefits in larger-scale post-training. These results, along with additional reward comparisons for HIL-SERL and RL-Token, have been added to the **Effects of Reward Design** subsection. Failures without human takeover are not directly supervised by the Failure Critic, while terminal task rewards provide complementary outcome-level feedback. **Q2:Hyperparameter Sensitivity and Operator Variability** We have added our previously conducted hyperparameter sensitivity analysis to the appendix. **Table.Hyperparameter Sensitivity Analysis** | Parameter | Value | Avg. SR (%) | |:---:|:---:|---:| | $K$ | 1 | 75.5 | | $K$ | **3 (default)** | **82.0** | | $K$ | 5 | 81.0 | | $\alpha$ | 0.025 | 79.0 | | $\alpha$ | **0.05 (default)** | **82.0** | | $\alpha$ | 0.1 | 77.5 | | $\beta$ | 0.25 | 76.5 | | $\beta$ | **0.5 (default)** | **82.0** | | $\beta$ | 1.0 | 78.0 | The results show that $K=3$ achieves strong performance and is selected as the default. On our platform, three policy steps correspond to approximately **3 s**. This window captures states immediately preceding human takeover, providing supervision near intervention boundaries. We agree that takeover thresholds may vary across operators. BELT-VLA estimates deployment-specific intervention risk rather than absolute failure probability, allowing adaptation to human intervention feedback during deployment. **Q3: Comparison with DexHiL and VLA-Pilot** We agree that DexHiL and VLA-Pilot are closely related. DexHiL uses recovery actions for policy training, while BELT-VLA uses pre-intervention states for risk supervision. Reproducing DexHiL requires adapting its training and data pipelines. VLA-Pilot relies on evolutionary diffusion, requiring adaptation to flow-matching architecture and contact-rich reward evaluation. Direct comparisons are not yet available due to implementation differences and time constraints. We are adapting both methods to the same π0.5/ALOHA platform and aim to include comparisons in the final revision. **Q4: Regarding Baselines and Simulation** We retain HIL-SERL's off-policy actor–critic framework and human-correction replay. RL-Token is reimplemented following the original method, with a frozen VLA and its original actor–critic optimization. We also add the same novelty–risk reward to both baselines. HIL-SERL drops from 77.5% to 69.0%, while RL-Token improves from 76.0% to 78.0%. This suggests that the effectiveness of reward shaping depends on the policy optimization mechanism. These results have been added to the Effects of Reward Design subsection. We agree that simulation benchmarks and cross-platform validation would strengthen our evaluation. We prioritized real-world experiments due to the challenges of simulating contact dynamics, friction, and perception noise, and plan to broaden our evaluation in future work. **Q5: Regarding Dependence on Human Intervention** We acknowledge that BELT-VLA still requires human supervision to identify initial failure boundaries. However, the human intervention rate decreases from 60% to **8%** during online adaptation, indicating improved autonomy and reduced reliance on human intervention. In future work, we will explore agent-in-the-loop mechanisms to reduce the need for continuous human monitoring, while addressing challenges in reliability and interaction overhead.

I thank the authors for their response. Their clarifications have addressed some of my concerns, and the additional ablation studies strengthen the empirical support for the work. However, my main concerns remain:

  • The method offers limited novelty, as it primarily combines existing techniques, including PPO and RND. The problem it addresses is also not particularly novel.
  • The reliance on human intervention in place of task rewards limits the method’s scalability.
  • The evaluation is limited to four real-world robotic tasks and includes no simulation experiments, raising concerns about whether the experimental assessment is sufficiently comprehensive.

For these reasons, I have decided to maintain my original rating.

Reviewer_2bxmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online adaptation of pretrained VLA models when deployment failures happen only in a few states. Instead of updating the whole policy, the proposed method BELT-VLA keeps base VLA frozen and edits only its latent action-flow features. A small edit policy outputs a steering vector, which is applied to the action-flow features. A Failure Critic is trained on human interventions to predict intervention risk. The edit policy is trained with PPO, and an edit penalty that is stronger in low-risk states. In the experimental results, success rate increases from 33.5% to 82.0% with proposed adaptation method, and the human intervention rate drops from 60% to 8%.

Critical strengths
  1. Studying deployment-time failures in VLA models via localized latent editing rather than global parameter updates is a well-motivated and practical direction for real robot deployment.
  2. The proposed methodology of constructing orthogonal transformations on latent action-flow features is a clean and reasonable way to adaptively update the featurs instead of global paramter update.
  3. The framework is validated on a real robot dual-arm platform across four challenging, contact-rich manipulation tasks, showing clear gains over offline baselines.
Critical weaknesses
  1. Some baseline comparisons may not be entirely fair. For instance, HIL-SERL and RL-Token are trained only with sparse terminal task rewards, whereas the proposed method receives dense per-step reward shaping. It is unclear whether the performance of these baselines would also increase if they were provided with the same dense rewards.
  2. The paper states that no task success reward is used during online adaptation. Without an explicit task success reward, it is unclear why the performance improvements are so significant. What is the main reason behind this? Furthermore, could adding a task success reward alongside human interventions improve the performance even further?
Questions for authors
  1. Since HIL-SERL and RL-Token are trained only with sparse terminal task rewards while the proposed method uses dense per-step reward shaping, how do these baselines perform if provided with the same dense rewards?
  2. Given that no task success reward is used during online adaptation, what is the main reason behind such significant performance improvements? Furthermore, could adding an explicit task success reward alongside human interventions improve the performance even further?
2 comments

We thank the reviewer for the positive assessment. We address the concerns about reward design and baseline fairness with additional experiments.

Q1: Reward Design and Baseline Fairness

To retain the original task-success rewards of HIL-SERL and RL-Token, we add the same novelty–risk reward to both baselines:

\qquad r_t = \alpha r_t^{cur} - \beta \rho_t.$$ The additional results are shown below. **Table. Performance Comparison under Different Reward Configurations** | Method | Reward Configuration | Avg. SR (%) | |:---|:---:|---:| | HIL-SERL | $r_t^{\mathrm{task}}$ | 77.5 | | HIL-SERL | $r_t^{\mathrm{task}} + r_t$ | 69.0 | | RL-Token | $r_t^{\mathrm{task}}$ | 76.0 | | RL-Token | $r_t^{\mathrm{task}} + r_t$ | 78.0 | | BELT-VLA | $r_t$ | 82.0 | | BELT-VLA | $r_t + r_t^{\mathrm{task}}$ | **83.5** | Adding the same novelty–risk reward slightly improves RL-Token (76.0% → 78.0%) but degrades HIL-SERL (77.5% → 69.0%). This suggests that its effectiveness depends on the policy optimization mechanism. suggesting that its advantage stems not only from reward design but also from localized policy editing mechanism, our editing ablations further support that. **Q2: Why Not Use Task Success Rewards?** When reproducing real-world RL methods, we observed that human recovery often leads to task success despite unreliable autonomous execution. Thus, terminal rewards may not reflect autonomous policy reliability. BELT-VLA instead uses intervention timing for step-level risk supervision, combined with RND novelty to guide localized policy adaptation. The pretrained VLA provides task priors, while RND encourages exploration and intervention risk discourages unreliable behavior. **Human intervention is not merely a corrective action; it also provides supervision on the reliability of autonomous behavior.** We agree that task success rewards may provide complementary supervision. We therefore add a sparse task reward to the original BELT-VLA reward: $$r_t^{\mathrm{BELT+task}} =r_t+r_t^{\mathrm{task}}.$$ Adding task rewards yields only a modest gain (82.0% → 83.5%), suggesting that explicit task-success rewards are not essential for effective online adaptation in our setting. We thank the reviewer for this valuable question, which may be particularly beneficial for larger-scale post-training. The experimental settings and results have been added to the **Effects of Reward Design** subsection.

I thank the authors for their detailed response. The rebuttal adequately most of my concerns. While I note the other reviewers' concerns regarding the incremental nature of combining existing algorithms, I maintain my positive evaluation and recommend acceptance (score: 3 / weak accept)

Comments on the paper

2 comments

Dear Reviewers,

I would like to kindly remind everyone to review the authors' responses and provide your feedback.

If you have any further questions or concerns regarding the authors' responses, please feel free to raise them early, so that the authors have sufficient time to address them before the rebuttal period ends.

Thank you all very much for your time and effort! And, have a nice weekend. :)

Best regards,

AC

AuthorsOpenReview ↗

Summary of Response and Manuscript Revisions

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful comments and constructive suggestions. We appreciate their recognition of BELT-VLA's motivation and practical value. We have addressed the main concerns through additional experiments, methodological clarifications, and manuscript revisions. We summarize the key points below.

1. Strengths Highlighted by Reviewers

We appreciate the reviewers' positive assessment of:

  • Localized policy adaptation. BELT-VLA addresses deployment-specific failures through local latent editing rather than global parameter updates.
  • Boundary-guided editing. The framework combines intervention-based risk estimation with state-conditioned orthogonal transformations to determine where and how to edit.
  • Real-world validation. Experiments on four contact-rich bimanual manipulation tasks demonstrate improved task success (33.5% → 82.0%) and reduced human intervention (60% → 8%).

2. Key Clarifications

  • Reward design without task-success signals. Human recovery often makes assisted episodes successful, yielding predominantly positive terminal rewards that provide limited supervision for unreliable autonomous behavior. BELT-VLA therefore uses intervention timing for risk supervision and RND for exploration, building on the pretrained VLA's task priors. Adding sparse task rewards yields a modest improvement, supporting their complementary value.

  • Reward design versus editing mechanism. We provide more detailed comparisons under different reward configurations in the new Effects of Reward Design subsection. Editing ablations further support the contribution of boundary-guided latent editing.

  • Scope of policy preservation. Our evaluation focuses on limiting policy deviations in reliable states within the adapted task. We acknowledge that cross-task evaluation is needed to establish broader capability preservation.

3. Additional Experiments, Analyses, and Revisions

  • Reward design comparisons (4.5 Effects of Reward Design, Table 3): Reward-augmented HIL-SERL and RL-Token baselines, and BELT-VLA with sparse task-success rewards.
  • Hyperparameter sensitivity (Appendix A.5, Table 6): Analysis of the intervention window KK and reward coefficients α\alpha and β\beta.
  • Failure Critic reliability (4.3, Appendix A.4): Held-out trajectory evaluation with an average AUROC of 0.82 and AP of 0.40.
  • Statistical analysis (Reviewer YKMZ): Additional 95% confidence intervals for success rates and comparisons against HIL-SERL.

We also acknowledge the value of broader comparisons. We are working to extend the evaluation with additional baselines and simulation benchmarks, and hope to include these results in the final revision.

We believe these additions strengthen the experimental evidence and clarify the contributions and scope of BELT-VLA. We sincerely thank the reviewers again for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.