acceptodds
Under review as a conference paper at ICLR 2027

Reflect, Recover, Internalize: Self-Improving VLAs from Execution Feedback

Abstract

Vision-language-action (VLA) models are usually trained from fixed demonstrations and discard the consequences and corrections observed during deployment. We introduce RRI-VLA, a reflect-recover-internalize framework that uses execution feedback for both immediate recovery and persistent improvement. At each action boundary, a frozen multimodal reasoner summarizes the preceding chunk as an outcome, running episode memory, and next adjustment. These states condition the next action, while a governed orchestrator combines semantic, physical, and learned evidence to authorize one bounded corrective chunk. An offline verifier then converts completed rollouts into verified failure-recovery records that train an action-conditioned Recovery Critic across deployments. After one-time feedback-interface initialization, only the critic is updated and the action-producing VLA remains frozen. On LIBERO-Plus and LIBERO-PRO, RRI-VLA reaches 92.47% and 57.84% task success. Relative to online-only recovery, internalization adds 3.12 and 5.64 points while retaining 95.30% success on unperturbed tasks; governed recovery resolves 74.2% of eligible failures with 1.10% harmful flips. Ablations identify verified experience, causal feedback alignment, and bounded governance as the key sources of improvement. Execution feedback thus provides a shared interface for within-episode recovery and policy-preserving learning across deployments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, failure recovery, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BichOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RRI-VLA, a framework that combines online failure recovery with cross-episode self-improvement for vision-language-action models. During execution, a frozen multimodal reasoner generates structured feedback from action outcomes, while a Recovery Orchestrator determines when corrective actions are needed. After each episode, an offline verifier identifies successful recovery experiences, which are used to update a Recovery Critic without further training the action-generating VLA. Experiments on LIBERO-Plus and LIBERO-PRO demonstrate improved task success rates and recovery performance under distribution shifts.

Critical strengths
  1. The paper combines online failure recovery with cross-episode self-improvement, forming a complete self-improvement loop. In addition to recovering from failures in the current task, the framework can accumulate failure and recovery experience for future improvement.
  2. The framework does not require continuous updates to the VLA. After initialization, only the Recovery Critic is trained.
  3. The experimental improvements are relatively significant. RRI-VLA achieves 92.47% on LIBERO-Plus and 57.84% on LIBERO-PRO, improving over the online-only version by 3.12 and 5.64 percentage points, respectively.
Critical weaknesses
  • Some important implementation details are still unclear. The paper describes what the verifier labels mean, but not exactly how they are computed. The initialization data and failure-to-instruction mapping also need more detail.
  • The experiments are limited to simulation. Appendix C.8 provides useful cost measurements, but the absolute latency and full-system delay on physical robots remain unclear.
  • Table 3 includes an external-reflection baseline, but comparisons with recovery methods such as ReCoVLA or AFI would better establish the contribution, with differences in supervision and sensing made clear.
Questions for authors
  • What signals and rules produce the recovery-value and failure labels?
  • What data are used for interface initialization, and how is the structured feedback constructed?
  • How are failure classes mapped to corrective instructions? Please also clarify the absolute reflection latency and total execution overhead.
Reviewer_rra3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RRI-VLA, a framework that uses execution feedback for both within-episode recovery and improvement across deployments. After each action chunk, a frozen multimodal reasoner summarizes what happened, what to remember, and what to adjust next. This summary conditions the next action, and a rule-based orchestrator may authorize one corrective chunk when needed. Completed rollouts are verified offline using privileged progress information from the simulator, and only a Recovery Critic, which decides when to trigger recovery, is trained iteratively on the verified records. The method is evaluated on LIBERO-Plus and LIBERO-PRO.

Critical strengths
  1. The formulation clearly separates what is updated in each round (only the critic) from what remains fixed, which allows the gains to be attributed to improved recovery decisions rather than changes to the motor policy.
  2. Interventions are localized to a single corrective chunk, so their outcomes can be attributed and verified, and the resulting records directly serve as training data.
  3. The paper includes many controlled ablations with careful designs, such as shuffled-feedback and shuffled-label controls, an oracle trigger, and a difference-in-differences analysis of feedback training.
Critical weaknesses
  1. Misleading title. The action-producing VLA is trained once during feedback-interface initialization and then kept frozen, and only the Recovery Critic is updated across rounds. The Abstract states this, but the title “Self-Improving VLAs” suggests that the VLA itself improves. A title that reflects the actual scope would be more accurate.

  2. Dependence on privileged simulator information. The offline verifier constructs its targets from simulator progress signals that are unavailable on a physical robot. The authors acknowledge this in the Limitations, but the main contribution, improvement from self-collected experience, depends on these signals. It is unclear how the framework would apply to real robots.

  3. Missing implementation details. The following details would be needed to understand and reproduce the method, and adding them to the appendix would strengthen the paper:

  • the model used as the Failure-Aware Reasoner
  • the list of failure classes and the corrective instructions g
  • the thresholds and values of the governance rules
  • the number of rollouts collected per round, and the size and training settings of the critic
  • the configuration of the External reflection baseline
  1. The paper provides no videos and few qualitative examples. Figure 1 shows a single illustrative sequence, but actual examples of the reasoner outputs, corrective instructions, and verified recoveries are not shown. Snapshots or videos of these steps would clarify how the framework behaves in practice.
Questions for authors

I am willing to raise my score if the followings are fully addressed.

  1. Please clarify the scope of the title, given that only the Recovery Critic is updated.
  2. How would the offline verifier be implemented without privileged simulator information?
  3. Please provide the implementation details listed in Weakness 3.
  4. F_n appears in Eq. 1 before it is defined in Eq. 6, and x_n is defined only in Eq. 8.
  5. Figures 1 and 2 appear to be raster images, and Figure 2 contains a typo (“Deply”).
  6. Could the authors provide qualitative results, such as videos or snapshot sequences of the recovery process? With detailed examples of how each components work?
Reviewer_8kEYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RRI-VLA, a VLA framework that uses execution feedback for failure recovery and self-improvement. The method includes a frozen multimodal reasoner that takes images, robot states, and actions as input and produces three outputs: outcome, episode memory, and next-step adjustment. The hidden states of these three outputs are used to condition the next action segment. A separate orchestrator uses this information to determine whether the instruction provided to the model should be modified. After a multi-round episode ends, the authors use privileged simulator states to label failures and recoveries. They then train an action-conditioned critic that determines when to intervene during inference. The authors compare their method with state-of-the-art VLAs on LIBERO-plus and LIBERO-pro, and they also conduct ablations of different components.

Critical strengths

None

Critical weaknesses
  1. Many implementation details are missing from the paper. These include which multimodal reasoner is used, its size, its prompt, the data and number of training steps used to teach the action experts to interpret feedback, the orchestrator’s parameter settings, the critic’s size and loss weights, and the evaluation settings. The authors also do not provide any qualitative results, such as the actual OUTCOME / MEMORY / NEXT outputs produced by the reasoner during inference, the failure cases detected by the critic, or the rewritten instructions. It is therefore impossible to determine how the action model uses this information to produce corrective actions.

  2. In this paper, the authors use an existing ACoT-VLA and freeze it after training. The only component updated during the entire process is a critic that determines when to modify the input instruction. Corrective actions are still predicted by the same frozen VLA. Therefore, if the frozen VLA has never seen a particular corrective action, it cannot predict that action. For this reason, the method cannot be considered truly self-improving. Collecting additional critic training data from rollouts using privileged simulator states also makes the comparison unfair. The baselines do not have access to this additionally collected data, so the improvement cannot be attributed entirely to the effectiveness of the method.

Questions for authors
  1. Which model is used as the frozen multimodal reasoner, how many parameters does it have, and what prompt is used? Which model is used as the VLM backbone of the VLA? Is the initial checkpoint based on the publicly released ACoT-VLA weights, or was it trained by the authors? What training parameters were used?
  2. What data and how many training steps were used to teach the action experts to interpret feedback? Does the training data include perturbations from LIBERO-Plus / LIBERO-PRO or rollouts from these benchmarks?
  3. In the appendix, the authors state that adding the reasoner / critic / orchestrator increases prediction time by only 12.2%. How was this 12.2% calculated? How many seconds does each component take to produce its prediction at each step?
  4. Can this framework operate on a real robot? The improvement across deployments relies on simulator progress as the label for the critic, but this signal is unavailable on a real robot. How could the framework be applied to real-robot settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.