acceptodds
Withdrawn from ICLR 2027

PermVLA: Factorization Order as a Regularizer for VLA Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies commonly learn action chunks through a fixed left-to-right (LTR) factorization, although the same expert trajectory distribution admits many valid chain-rule factorizations. We identify factorization order as an overlooked regularization choice and introduce causally anchored permutation (CAP), which samples action reveal orders with a tunable chronological prefix. Its auxiliary objective trains one shared policy to predict actions from different known subsets of the same expert chunk, while deployment retains deterministic LTR control. We call this *conditional-set augmentation*: it creates multiple conditional prediction problems from one expert chunk without adding demonstrations. This discourages reliance on the single chronological prefix used by ordinary teacher forcing. Controlled experiments show that CAP consistently outperforms standard LTR training on LIBERO and LIBERO-Plus, with the same advantage appearing in cross-dataset CALVIN evaluation. A diagnostic that measures the expected squared difference between a chunk's joint log likelihood under two reveal orders verifies that CAP training internalizes agreement across reveal orders. These findings position sampled subset-conditioned auxiliary objectives as a general recipe for constructing VLA regularizers, illustrated by an extension to diffusion action generators.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Out-of-Distribution Generalization, Autoregressive Factorization, Robot Learning, Regularization, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rCQCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PermVLA, a method import permutation language modelling into VLA. Unlike autoregressive prediction, PermVLA creates subsets from an action chunk with permutation for training. The experiments are carried out on LIBERO dataset. Additional evaluation for zero shot is performed on LIBERO-Plus and cross task evaluation on CALVIN.

Critical strengths
  • Compared to autoregressive modelling, PermVLA shows improvement on LIBERO, LIBERO-Plus and CALVIN.
Critical weaknesses
  • The permutation ignores the temporal dependencies between actions, this potentially hurts performance on long-horizon tasks.

  • The experimental results are not convincing. Multiple baselines (pi0, SmolVLA, WorldVLA and RIPT-VLA) outperforms PermVLA on LIBERO. The strongest baseline i.e. RIPT-VLA outperforms PermVLa around 10% on spatial, object and goal tasks in LIBERO, and is around 60% on long horizon task. Same on the zero shot setting, RIPT-VLA significantly outperforms PermVLA. On the cross task setting, OpenVLA significantly outperforms PermVLA on all metrics.

Questions for authors

What is the motivation to use permutation modelling given the actions have temporal dependencies and they are critical?

Reviewer_eo99OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PermVLA, which introduces factorization order as a regularization strategy for VLA learning. The proposed Causally Anchored Permutation (CAP) objective trains an autoregressive action policy using different action reveal orders while preserving chronological action generation during inference. This allows multiple conditional prediction tasks to be constructed from the same expert demonstrations. Experiments on LIBERO, LIBERO-Plus, and CALVIN show improvements over standard left-to-right (LTR) training, particularly on long-horizon tasks.

Critical strengths

Simple and well-motivated method. The paper explores an interesting perspective on VLA training by treating action factorization order as a regularization choice. CAP is easy to incorporate into autoregressive training and does not require additional demonstrations or changes to inference.

Useful controlled experiments and ablations. The comparisons with LTR-1pass and LTR-2pass help distinguish the effect of CAP from additional decoder computation. The ablations on permutation settings and the factorization sensitivity analysis provide some evidence supporting the proposed design.

Critical weaknesses

Insufficient comparisons with relevant and competitive baselines. The main experiments only compare CAP with LTR-1pass and LTR-2pass. While these controls help isolate the effect of additional computation, they do not establish whether CAP is more effective than existing permutation-based or masked action modeling objectives. The lack of such comparisons makes it difficult to assess the specific contribution of CAP beyond general conditional-set augmentation. Moreover, the K=0 configuration achieves comparable average performance to K=1, leaving the benefit of the proposed causal anchor unclear.

Insufficient experimental evidence for generality and robustness. The main improvements are demonstrated on a custom Gaussian autoregressive policy. Although the appendix includes backbone replacement and PermDiffusion experiments, neither provides corresponding matched baseline comparisons, so the effectiveness of CAP on these alternative architectures remains unverified. Furthermore, the factorization sensitivity analysis does not directly demonstrate improved robustness to execution errors or recovery. Appendix D also reveals limitations in the task success criteria, suggesting that additional evaluation of stable and physically complete task execution would strengthen the conclusions.

Questions for authors

Can the authors compare CAP with alternative masked action modeling, random subset completion, or permutation-based training objectives under the same architecture and computational budget? In particular, what is the specific benefit of the causal anchor compared with unrestricted permutations (K=0)?

The H=8 configuration achieves the highest average LIBERO success rate (77.5%), but the main LTR baselines use H=16. Can the authors provide matched LTR comparisons under H=8 to better establish the improvement attributable to CAP?

Appendix B reports PermDiffusion results without a standard diffusion baseline, while the PaliGemma backbone experiment also lacks a corresponding LTR control. Can the authors provide these matched comparisons to verify that CAP improves performance beyond the original autoregressive architecture?

Can the authors provide more direct evidence that CAP improves robustness during closed-loop execution, such as controlled perturbation and recovery experiments? Additionally, given the success-predicate limitations identified in Appendix D, do the improvements remain under stricter criteria requiring stable object placement and gripper release?

Reviewer_hin4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses causally anchored permutation as an auxiliary training objective for a continuous Gaussian AR VLA. Basically, the policy predicts action chunks under different reveal orders while retaining regular chronological generation in test time. Experiments are mainly on LIBERO, LIBERO-Plus, and CALVIN. This paper also studies agreement across factorization orders and a preliminary diffusion extension of the proposed method.

Critical strengths

Main pros: The proposed intervention is very simple to implement. The method constructs additional conditional prediction problems from existing demonstrations while preserving the deployment procedure.

Critical weaknesses
  1. Experiment setting is very limited. No real-world results. Results focus on very saturated LIBERO and LIBERO-Plus. Moreover, performance is in general weaker than existing VLA policies with flow action heads.

  2. The algorithm setting and motivation are also limited. The proposed method only makes sense to action-wise AR, i.e., chronological factorization over individual action timesteps has to be a property of the chosen head. Diffusion/flow policies that directly predict the whole action chunk don't impose the same structure and they don't have the issue mentioned in this paper at all. Similarly, AR + chunk-level tokenizations don't impose the same structure either. Chunk-level tokenizations already encode dependencies across physical timesteps. The paper lacks matched comparisons with these alternatives. To some extent, this paper is like "finding a better solution in a suboptimal setting".

  3. Training variability is not quantified. The main experiments use a single training seed. The large number of evaluation rollouts does not resolve uncertainty across independently trained policies.

Questions for authors

See the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.