acceptodds
Withdrawn from ICLR 2027

RouteRL-VLA: Learning from Failures via Factorized Routing for Vision-Language-Action Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models can acquire broad manipulation skills through large-scale pretraining and supervised fine-tuning. Yet a failed rollout is reduced to a single outcome signal. Without knowing where the error arose or which part of the policy caused it, existing reinforcement-learning methods for VLA models often spread credit across semantic, visual, and execution failures, diluting informative gradients and risking interference with previously acquired capabilities. We introduce RouteRL-VLA, a failure-aware post-training framework that factorizes credit over failure type, trajectory time, and parameter group. It constructs and compares paired clean and perturbed rollouts, uses a selective paired teacher to estimate failure attribution, soft failure-type probabilities, and query-level temporal weights, and directs the resulting advantages to specialized policy adapters. For stable on-policy learning with continuous action chunks, we develop joint action-chunk likelihoods, duration-aware generalized advantage estimation, and transactional KL-constrained PPO. When applied to OpenVLA-OFT on LIBERO-Goal, RouteRL-VLA attains SOTA failure-recovery performance among five trained methods: 62.15% perturbed macro success, 2.99 points above uniform credit, with 95.83% clean success. In a separate 1,224-episode evaluation, it improves success from 45.26% to 78.35%, a 33.09-point gain. Oracle routing and route-permutation controls confirm that accurate route-failure alignment drives this improvement. The attribution model reaches 0.988 AUROC, while the selective router achieves 0.992 macro-F1 at 0.943 coverage. The experiments jointly show that paired diagnosis identifies how a rollout failed, factorized routing targets the responsible update, and alignment controls verify that informative routes drive recovery.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, Robot Manipulation, Reinforcement Learning, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_FXroOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RouteRL-VLA is an RL post-training framework for Vision-Language-Action (VLA) models that factorizes credit assignment across failure type (Semantic, Visual, Execution), trajectory time, and three LoRA adapter groups on a frozen VLA backbone. It uses paired clean and synthetically perturbed rollouts from the same task, initial state, and simulator seed to train a selective teacher that estimates failure attribution, failure-type probabilities, and temporal weights for chunk-level PPO updates. The authors validate the method on OpenVLA-OFT using the 10-task LIBERO-Goal suite under synthetic perturbations.

Critical strengths

Trajectory-level binary rewards in VLA RL post-training fail to distinguish whether a failure arose from language comprehension, visual perception, or motor execution, and that on-policy RL updates should operate at the joint action-chunk level with duration-aware GAE identifies a practically relevant credit-assignment problem in post-training VLA policies.

Critical weaknesses
  1. Undefined perturbations and failure types (L141, L149–151): The entire method and evaluation rely on three synthetic perturbation types (sem, vis, exe). Beyond a single generic sentence in L149–151, the paper never specifies how instructions are altered for sem, what visual corruptions are applied for vis, what control-channel disturbances are injected for exe, or what the "perturbation schedule" (L219) is. Consequently, "failure type" is merely the synthetic perturbation label injected by the authors rather than an intrinsic failure mode of the policy.

  2. Undefined "Selective Paired Teacher" and vague notation (L145, L155–161): In L145, "The shared context" is referenced without clearly defining it. In L155, τ0\tau^0 and τc\tau^c are used without subscripts or clarification of whether they represent full post-hoc trajectories or partial histories up to query qq. Crucially, the paper never defines what features ϕ(τ0,τc)\phi(\tau^0, \tau^c) are extracted, what the teacher's architecture or training loss is, how the ground-truth "divergence region" for temporal weights wqw_q (L160, Table 8) is mathematically defined or supervised, or what the abstention thresholds are.

  3. Unjustified need for a learned teacher router (L141–161, L318): At test time, the teacher router is not used at all, nor can it be used in real-world deployment since it requires a paired clean reference trajectory τ0\tau^0 from the exact same initial state and simulator seed. Conversely, during simulated training, generating the pair (τisz0,τiszc)(\tau_{isz}^0, \tau_{isz}^c) already requires explicitly injecting a known perturbation c∈{sem,vis,exe}c \in \{\text{sem}, \text{vis}, \text{exe}\} and observing the terminal outcomes y(τisz0)y(\tau_{isz}^0) and y(τiszc)y(\tau_{isz}^c) (Eq. 1). Because both the perturbation type and attribution outcome are already known by construction in simulation and unavailable at test time, it is unclear why a learned teacher classifier is needed at all.

  4. Temporal misalignment during on-policy PPO updates (Figure 1, Eq. 3): If the paired trajectories (τ0,τc)(\tau^0, \tau^c) are collected upfront and their teacher outputs are reused for subsequent "fresh action-chunk rollouts" (Figure 1) during PPO training, the temporal credit assignment becomes invalid. As the policy πθ\pi_\theta is updated by PPO, the actions and trajectories of new rollouts change at every iteration; thus, multiplying a new rollout's advantage at step qq by the temporal weight wqw_q computed from an earlier trajectory (Eq. 3) is misaligned in time. Conversely, if a new paired clean rollout τ0\tau^0 must be executed online for every perturbed RL rollout, the simulation cost doubles and requires exact simulator state/seed resets.

  5. Unfair baseline comparison in the 1,224-episode evaluation: In Table 5, the RL post-trained RouteRL-VLA (78.35%) is compared only against the un-finetuned base model OpenVLA-OFT, omitting the RL baselines (Vanilla PPO and Uniform credit). Comparing an RL-finetuned policy against a zero-shot base model on the perturbed distribution does not isolate the benefit of the proposed routing mechanism.

  6. Narrow evaluation on a single model and a single sub-suite: Standard evaluation requires testing across all four LIBERO benchmark suites (LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, LIBERO-Long/LIBERO-10), as well as broader benchmarks such as RoboCasa, SimplerEnv, or public perturbation benchmarks like LIBERO-Para (which the authors cite in L111 but do not evaluate on). Evaluating only a single base model (OpenVLA-OFT) on a single 10-task suite (LIBERO-Goal) under undisclosed custom perturbations is insufficient to support the paper's claims.

  7. Manuscript preparation and presentation issues: Out of the eight tables in the paper, only Table 2 is ever referenced in the main text, and Tables 1, 3, 4, 5, 6, 7, and 8 are completely uncited in the text. Furthermore, the tables are numbered out of order (Table 7 appears on page 7 before Tables 5 and 6 on page 8), Table 6 reports qualitative text ("Upper bound passed") without any numerical values, and there are multiple formatting typos.

Questions for authors

Following the weaknesses outlined above, my primary concern is that the current manuscript lacks essential details needed to fully understand and reproduce the proposed method, alongside concerns regarding the methodological motivation, experimental breadth, and overall manuscript presentation, which led to my current scores. I would appreciate the authors' clarification on the points raised above.

Reviewer_xUC9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RouteRL-VLA post-trains a frozen OpenVLA-OFT. Each clean rollout is paired with a perturbed one that changes a single factor (semantic, visual or execution). A selective paired teacher outputs an attribution gate, a failure-type posterior and temporal weights. These scale a chunk-level PPO advantage that is routed to one of three LoRA adapters. The method also uses a joint action-chunk likelihood, duration-aware GAE and KL-triggered rollback. On LIBERO-Goal, perturbed macro success is 62.15% vs. 59.16% for uniform credit (Table 2), and on a 1,224-episode evaluation success rises from 45.26% to 78.35% over the base model (Table 5).

Critical strengths
  1. The motivation holds: a single trajectory-level advantage spreads credit over failures with different causes. Matching clean and perturbed rollouts on task, initial state and seed makes failure attribution observable.
  2. The chunk-level PPO implementation is careful (exact joint likelihood, duration-aware GAE, float64 checks, rollback of the full optimizer state). These details are useful for anyone doing RL on VLAs.
Critical weaknesses
  1. The main gains are within noise, and no variance is reported. The Table 2 percentages imply small cells (e.g., 35/42 vs. 34/42, 72/108 vs. 66/108), so the +2.99 pp over Uniform amounts to about 8 extra successful episodes. Three seeds are mentioned, but no std or test is given, and ablation gaps of 0.86 to 2.49 pp cannot support the claim that every factor is necessary. Vanilla PPO also has the highest clean success (98.33 vs. 95.83).
  2. Several comparisons are mis-specified or reported without numbers. The 33-point gain in Table 5 is over the untrained base model, not over PPO or Uniform, so it does not isolate routing. Table 6 gives only qualitative outcomes. The oracle described as an upper bound scores below the method in Table 2 (60.38 and 58.62 vs. 62.15), and Oracle Type-Time is below Uniform; the paper does not explain either result. Evaluation is limited to LIBERO-Goal with no comparison to SimpleVLA-RL or VLA-RL, and the teacher is trained on 140 examples.
Questions for authors
  1. Please report per-seed results, mean ± std, per-cell episode counts and paired significance tests for Tables 2 and 4.
  2. Please run the Table 5 evaluation for Vanilla PPO and Uniform at the same budget, and explain where the 45.26% base success rate comes from.
  3. Please give numbers for Table 6. Why does the oracle score below the method, and why is Oracle Type-Time below Uniform?
  4. The perturbation type is known by construction during training. What does the learned teacher add beyond that label? Does the method apply when no clean counterfactual is available?
  5. Please compare with SimpleVLA-RL and VLA-RL and add at least one more benchmark. The appendix referenced in the text is also missing.
Reviewer_PSBvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a failure-aware VLA RL post-training method using paired "clean" and "perturbed" rollouts. A selective teacher routes advantages by failure type, trajectory time, and adapter group while the backbone remains frozen. The main claim is that this method has better credit assignment compared to regular RL post-training. Empirical study is done in LIBERO-Goal.

Critical strengths

The proposed factorization into failure type, trajectory time, and parameter groups provides an intuitive way to structured credit assignment for VLA RL post-training

Critical weaknesses
  1. The experiment setting is too limit. Only LIBERO-Goal sim bench is considered, which is already very much saturated. Only one model OpenVLA-OFT is considered. Moreover, how to implement the proposed perturbation method in the real world?

  2. A lot of technical details are very unclear, for example, teacher inputs/supervision, adapter placement and composition, and parameter-specific gradient routing. These omissions make it hard to fully understand how the proposed method works.

  3. The strongest mechanism claims (in table 6) needs quantitative support. The current table 6 doesn't have any numerical scores.

Questions for authors

See the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.