acceptodds
Under review as a conference paper at ICLR 2027

Learning from Runtime Feedback with Failure-Bank Self-Evolution for Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models generalize broadly across robotic manipulation tasks, but complex environments require balancing task success with unintended contact. Runtime shields can correct individual actions, but they leave the underlying policy unchanged, so repeated disagreements may create a persistent policy–shield mismatch that blocks task progress. To address this challenge, we introduce FAILBANK, a four-stage self-evolving framework that converts runtime feedback into persistent policy improvement. During collection, a fixed CBF-based safety module serves as an observe-only teacher, producing counterfactual corrections while the policy remains in control. Outcome-aware admission then converts useful proposals into corrective targets and retains successful uncorrected actions as quiet anchors for guarded LoRA updates. We evaluate FAILBANK on the VLA-Arena benchmark across two difficulty levels and two VLA backbones. Compared with the base policies, FAILBANK improves the joint success–cost operating point. Across the two backbones, FAILBANK improves task success rate by 8.5 and 6.9 percentage points, while reducing policy-induced cumulative cost by 35.6% and 23.8%, respectively. Compared with runtime shielding, FAILBANK raises task success rate by 25.4 and 9.5 percentage points, while maintaining comparable policy-induced cumulative cost. These results show that runtime feedback can serve as persistent policy supervision rather than only as a temporary action constraint.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Safety, Runtime Feedback, Self-Evolving Policies, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hMXzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FAILBANK, a framework that turns runtime safety feedback into lasting policy improvements for vision-language-action (VLA) models. It addresses a key weakness of runtime shielding: while a shield can correct unsafe actions during execution, it does not update the underlying policy. As a result, the same mismatch between the policy and the shield can recur over multiple steps, reducing task completion performance. FAILBANK resolves this issue by using a fixed control barrier function (CBF)-based safety module as an observe-only teacher during rollout collection. The policy continues to act normally, while the teacher records counterfactual corrective actions whenever safety intervention would have been needed. These corrections are then filtered based on rollout outcomes, combined with successful uncorrected actions as quiet anchors, and stored in a failure bank. The accumulated data is then used to perform guarded LoRA updates on the original policy.

Critical strengths

A major strength of the paper is that it highlights an important and realistic failure mode of runtime safety systems: the persistent mismatch between a nominal policy and an external shield. The core idea of converting runtime corrections into training supervision is compelling, intuitive, and well motivated. The method is presented clearly, with a four-stage pipeline that is easy to follow and conceptually consistent. The experimental section is another strong point. It evaluates the approach across two backbones, two difficulty levels, and comparisons with both the base model and AEGIS. The paper also provides useful ablation studies that support key design decisions, particularly the choice of observe-only collection over shield-in-the-loop collection and the inclusion of quiet anchors to stabilize learning. A further practical advantage is that the adapted policy can be deployed without access to privileged geometry or a runtime shield, making the approach more appealing than methods that rely on continuous safety intervention at test time.

Critical weaknesses

Most of the evidence in the paper comes from the static-obstacle setting, so the conclusions are somewhat limited in scope. The results on dynamic obstacles are weak or unclear, which suggests that the method may not yet work well in harder safety scenarios. The approach also relies on privileged geometric information during data collection. While this makes sense in simulation, it raises questions about how well the same process would transfer to real-world robots. The comparison with AEGIS is useful in practice, but it is not a perfectly fair comparison. FAILBANK uses privileged teacher signals during training, while AEGIS depends on runtime grounding and perception. Another issue is that the method includes several hand-crafted design choices, such as admission rules, weighting bins, and guard thresholds, which may make it sensitive to implementation details. Finally, the paper focuses mostly on empirical results and would be stronger with a deeper explanation of when and why observe-only corrections lead to reliable policy improvement.

Questions for authors

First, it would be helpful to know how sensitive FAILBANK is to the quality of the CBF teacher. If the teacher is noisy, too conservative, or biased in some systematic way, does the method still improve the policy, or can it end up learning bad corrections? Second, the paper should explain more clearly how the privileged data collection setup could be replaced in a real robot setting, for example by using learned perception or rough geometry estimates. Third, since the method uses several thresholds and heuristic design choices, it would be useful to know which ones matter most and how stable the results are when they are changed. I am also curious why the main training signal uses only first-action supervision instead of full action chunks, and whether this choice is important for stability. In addition, a stronger comparison with simpler finetuning baselines would help show how much of the improvement really comes from the failure-bank design, rather than just retraining on failure cases or successful trajectories. Finally, the paper would benefit from more discussion of whether the method can be extended easily beyond flow-matching policies to token-based or diffusion-based VLA models.

Reviewer_fkBROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FAIL BANK, a framework that uses runtime CBF feedback to improve VLA policies. It collects counterfactual action corrections, filters and accumulates training records, and updates the policy through guarded LoRA fine-tuning. Experiments on VLA-Arena with two VLA backbones demonstrate improvements in task success and reductions in policy-induced cumulative cost relative to the base policies.

Critical strengths

The paper addresses the policy–shield mismatch in runtime safety control. Learning from CBF corrections instead of repeatedly applying them during execution is a reasonable idea. The improvements across two VLA backbones demonstrate the potential of this approach.

Critical weaknesses

1. Limited algorithm novelty. The framework mainly combines existing techniques, including CBF-based action correction, DAgger-style supervision, data filtering, and LoRA fine-tuning. Although several attribution experiments are provided, direct comparisons with standard CBF-guided DAgger or corrective imitation learning baselines are missing. This makes it difficult to assess the benefits of the proposed framework over simpler teacher–student approaches.

2. Limited real-world applicability and experimental validation. The experiments are entirely simulation-based, and the CBF teacher relies on privileged obstacle geometry that is not directly available on real robots. Moreover, the observe-only collection strategy executes potentially unsafe actions without intervention, raising concerns about real-world feasibility. The evaluation is also largely restricted to static obstacles, with inconclusive results on dynamic-obstacle transfer. These limitations weaken the practical significance of the proposed framework.

Questions for authors
  1. How does FAIL BANK compare with a standard CBF-guided DAgger or corrective imitation learning baseline under matched data and training budgets?

  2. How could the observe-only collection strategy be safely implemented on real robots? How would errors in estimated obstacle geometry affect the quality of the teacher supervision?

  3. What are the independent contributions of failure-bank accumulation and quiet anchors? In particular, why is the quiet-anchor weight set to zero in the main π0.5 experiments?

Reviewer_qkjtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The key idea of this paper is to turn a safety controller’s proposed corrections into training data for a VLA policy. The proposed framework, FAILBANK, adapts VLAs using feedback from a CBF (Control Barrier Function) teacher. During data collection, the policy executes its nominal actions while the teacher logs proposed corrections using privileged geometry in simulation (but the corrections are not executed). Selected roll out data accumulate in a bank and supervise LoRA updates, which are accepted using held-out loss and drift checks. The adapted policy is deployed without the teacher. Experiments are done on VLA-Arena sim bench with both \pi_0 and \pi_0.5 models.

Critical strengths
  1. Sensible approach with empirical gains on VLA-Arena. The key idea, collecting unexecuted teacher proposals, preserves the policy’s rollout distribution and enables adaptation without requiring a shield at deployment.

  2. Substantial ablations and diagnostic evaluations. The zero-residual, randomized-correction, and successful-trajectory controls help isolate the contribution of corrective supervision.

Critical weaknesses
  1. The experiment setting is too simple. No real-world experiments, only VLA-Arena is considered in sim. Moreover, the proposed pipeline (using a CBF teacher) is not tractable in the real world.

  2. The robustness of the reported improvement is not sufficiently established. The proposed framework FAILBANK achieves higher success at higher mean policy-induced cost on both backbones. Dynamic-obstacle experiments do not establish positive transfer.

  3. How does the complete iterative procedure compare with a single round of filtered corrective finetuning under matched total collection and optimization budgets? The existing accumulation controls are informative but cover only a subset of tasks and do not establish a consistent success benefit.

Questions for authors

See the weakness part.

Comments on the paper

1 comment

Dear Reviewers,

Thank you so much for your time and constructive feedback. With the discussion deadline coming up, we’d appreciate any thoughts on our rebuttal and additional experiments. We’d be happy to answer any remaining questions and would welcome suggestions for revising the paper.

Thanks again for reviewing our work!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.