acceptodds
Withdrawn from ICLR 2027

When Scenes Shift: Transition-Referenced Adaptation for Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies can degrade substantially under scene shifts even when task-relevant state evolution retains a similar structure. In few-shot adaptation, target demonstrations directly supervise *what to do* in the new scene, but provide limited guidance on how shifted observations should reconnect to the policy's existing control representations. Motivated by these observations, we introduce **ShiftVLA**, a transition-referenced framework that treats the relation between internal control and task-relevant state evolution as a reference for few-shot scene adaptation. ***(i)*** To construct this reference, a transition learner captures demonstrated state progression through intermediate-state prediction and aligns transition relations with action-chunk similarity. ***(ii)*** A lightweight reader maps depth-aggregated Action-Expert control states into the transition space, while joint grounding shapes the underlying control representations with both transition and native action supervision. ***(iii)*** During target adaptation, source-calibrated control readouts drive depth-wise refinement, while gradient projection suppresses action-update components that conflict with the established correspondence. Under low-shot adaptation, **ShiftVLA** improves average success over the strongest baseline by 1.8 and 3.5 percentage points on LIBERO-Plus and RoboTwin 2.0, respectively, and improves real-robot task success by 7.0 percentage points under controlled scene shifts.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Few-Shot Adaptation, Robot Manipulation, Scene Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_F4hSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the problem of finetuning VLAs under scene shifts. Instead of directly finetuning to data collected under the new scene, the paper proposes to learn an invariant representation of the robot's state change, called "execution transitions." They then establish a correspondence between the "execution transtions" and the latent representations in the VLA's Action Expert. This correspondence is then used to guide the finetuning of the Action Expert. The finetuning involves encoding a few "prototype" source domain trajectories into a coordinate, which is used to guide the LoRA weight updates to the target domain. The paper evaluates on two simulation benchmarks, LIBERO-Plus and RoboTwin, each grouping tasks into source and target domains. The proposed ShiftVLA achieves the highest overall performance across both benchmarks in the few-shot setting. The paper further evaluates on a real GALAXEA robot platform, showing more efficient finetuning under scene shifts compared to OpenVLA and Pi-0.5.

Critical strengths
  1. The paper targets the imporant problem of efficient finetuning across domain shifts.
  2. The real robot experiments seem impressive.
Critical weaknesses
  1. The method is unnecessarily complex. They first establish a correspondence between the robot's state change and the VLA action expert's hidden states. The state change, or "execution transition," is captured by an encoder that takes in the initial and final image observations of a trajectory segment, Zt=Fϕ(Ev(It),Ev(It+H))Z_t = F_\phi(E_v(I_t), E_v(I_{t+H})). It is trained to (1) reconstruct intermediate imaged features conditioned on some progress u∈[0,1]u \in [0, 1], (2) keep the distance between the ZZs closely distributed to the distance between the actions aa. (this doesn't quite make sense, as they defined a distribution proportional to distance and then minimize their KL divergence, see Eqn 5, 6). Then, having trained this delta state encoder, they train another mapping from the VLA's action expert's latent states to the delta state space. During finetuning, they first construct a "transition-reference coordinate" from a couple of prototype transitions from the source domain. This coordinate is then injected to the LoRA finetuning for guidance. Finally, the delta state representation is also used to shape the gradient update so that only correspondence-preserving gradient directions are kept.
    • My high-level understanding of the algorithm is: train a joint representation space where the state change and the corresponding action chunk are mapped close to each other. Then finetunine the VLA action expert while preserving this correspondence.
    • The ablations in Table 4 only covers a small subset of the algorithm's design choices. There are between 10-20 design choices, most of which seem arbitrary.
  2. Following from the first point, the writing is extremely terse and borderline incoherent. The entire paper is littered with undercontextualized jargons. A few examples:
    • Lines 47-49: "We instantiate this view in SHIFTVLA, which learns a transition reference, establishes its correspondence with Action-Expert control states, and organizes few-shot scene adaptation around the resulting control–transition correspondence." The paper does not provide enough context for the reader to understand what "transition reference," "Action-Expert control states," or "control-transition correspondece" mean.
    • Lines 75-77: "Source-calibrated control readouts modulate low-rank control refinement across Action-Expert depth, while gradient projection removes action-update components that would weaken the grounded correspondence." This is extremely confusing.
    • Lines 166-167: "We denote by dLN(X,Y)=mean∣LN(X)−LN(Y)∣d_{LN}(X,Y) = mean|LN(X)-LN(Y)| the distance between layer-normalized features." is completely out of place. This is not used until several lines later, and when I got there it felt like dLNd_{LN} was never introduced.
    • This can also be seen from Figure 2, which is extremely dense and hard to understand.
    • Overall, this paper needs signficant overhaul to reach a state that is ready for peer-review. This is beyond the scope of a rebuttal.
Questions for authors
  1. Can you provide a brief description of each baseline being compared to, e.g. FLA, Robotic Sttering, and RETAIN?
  2. Can you compare to efficient finetuning baselines on the real robot? The simulation results show slight improvement over effcient-finetuning baselines like FLA and RETAIN. It's unclear if this trend holds on a real robot.
  3. Can you provide a more comprehensive ablation of the algorithms individual design choices? E.g. the use of KL divergence for Lrel\mathcal L_{rel}?
Reviewer_gpqfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • ShiftVLA proposes a framework for few-shot adaptation of vision-language-action (VLA) policies, aiming to tackle scene distribution shifts.
  • Instead of adapting exclusively through action supervision on target demonstrations, the approach leverages the pretrained policy's internal representations mapped to a learned representation of demonstrated execution transitions.
  • The method uses a modular approach to: a) learn the transition reference, b) ground the control transition, and c) train with correspondence guidance.
  • The method is evaluated on LiberoPlus, RobotTwin 2.0, and GALAXEA R1.

Claims:

  1. Execution transitions provide a useful reference for scene adaptation.
  2. Grounding transition representations in Action-Expert states provides a useful control interface.
  3. Correspondence-guided refinement and gradient projection improve few-shot adaptation.
Critical strengths
  • Clear motivation, about keeping the demonstration state-progression form policy’s generative action representation. Transition learning establishes references, grounding combines it with action-expert and then training/adaption using the correspondences.
  • Decent experimental coverage. LiberoPlus is especially suited for this problem statement. The evaluations indicate this approach show’s promise as its not restricted to a single perturbation type (across benchmarks).
  • Tab.4 Ablations is thorough, with the three components providing the most gains.
  • The action–transition similarity analysis reports correlations of 0.5939 for within-condition pairs and 0.5860 for cross-condition pairs. The authors also report stronger geometric agreement between Action-Expert representations and the transition reference than between VLM representations and the reference
Critical weaknesses

-The central representation-level claim is not fully isolated from the action-relation supervision. The main conceptual contribution is the claim that a transition reference provides a useful structural anchor for adaptation. However, the transition representation is explicitly trained to align its neighborhood geometry with action-chunk similarity. The paper fails to distinguish between demonstrating that a reference is useful for adaptation and demonstrating that it captures a distinct, transferable representations.

  • The empirical improvements are consistent but often modest. While three support-set samples provide some robustness to support-set selection, it is difficult to determine whether the smaller improvements are statistically meaningful or within the noise margin.

  • The method introduces a lot of complexity, a transition encoder, state predictor, reader, depth aggregation module, and gradient projection during adaptation. Furthermore, the work lacks evaluations on scenarios that alter transition dynamics apart from visual and environmental appearance, e.g. change in object shape.

Questions for authors
  1. Can the authors compare against a parameter-matched low-rank adapter with comparable depth-wise conditioning but no transition reference.
  2. Can the authors replace the learned transition reference with other baselines like action-space or generic feature-space targets.
  3. Computation overhead of ShiftVLA, Can the authors report target-adaptation time, inference latency, memory usage, and the additional computation per denoising step relative to standard low-rank adaptation ?
  4. Compute Fairness ShiftVLA’s source policy receives 20k additional joint-grounding steps plus a separately trained 30k-step transition model, while baselines appear to adapt from the released π0.5 checkpoint. A controlled experiment that continues π0.5 training on source data for a matched budget is needed. [The “source.” column improving from 95.7 -> 96.9 might indicate the gain is simply extra source training.]
Reviewer_7RSeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to adapt vision-language-action policies when familiar tasks are performed under different scene conditions, such as changes in camera viewpoint, lighting, background, or object layout. Adaptation uses a small number of demonstrations from the new conditions.

ShiftVLA learns a representation of how a scene changes from the first and last images of short demonstration segments. Training encourages this representation to predict visual features at intermediate times and to be similar for segments with similar action sequences. A learned reader maps the internal states of the π0.5\pi_{0.5} action expert to this representation. During adaptation, these predictions adjust low-rank adapter activations, while gradient projection modifies action-learning updates that conflict with the transition-matching objective.

The experiments cover LIBERO-Plus, RoboTwin 2.0, and five real-robot tasks. Under five-shot adaptation, the method improves average success over the strongest reported simulation baseline by 1.8 percentage points on LIBERO-Plus and 3.5 points on RoboTwin. Real-robot task success increases from 59.0% to 66.0% relative to π0.5\pi_{0.5}.

Critical strengths
  • Using the scene changes shown in demonstrations to guide adaptation is a sound idea. ShiftVLA connects these changes to the policy’s internal action representations and uses that connection during both action computation and parameter updates.
  • The method improves average success in both simulation benchmarks and on the real robot. The ablations show that the transition-matching loss, adapter conditioning, and gradient projection each improve success within the proposed framework.
Critical weaknesses
  • The paper does not adequately justify omitting adaptation on mixed source and target data. The experiments largely involve scene variations of familiar tasks, and the original demonstrations are available and used extensively to train ShiftVLA. A natural alternative is therefore to train low-rank adapters on a mixture of the original demonstrations and the new demonstrations. Table 4 includes action-only low-rank adaptation, but does not describe a baseline that also uses the original data during adaptation. Please clarify whether this comparison was already performed.
  • Several quantities needed to implement the method are not sufficiently defined. In Section 2.4, γℓ\gamma_\ell controls how strongly the transition representation changes the adapter activations. The paper describes it as source-derived, but I could not find how it is computed. The same section introduces source prototypes and the means μℓbc\mu_{\ell bc} and variances vℓbcv_{\ell bc} used in Eq. 10, but does not clearly explain how the prototypes are constructed or which source samples contribute to each set of statistics.
Questions for authors
  • How does ShiftVLA compare with low-rank adaptation on mixed original and new demonstrations, using a reasonable sampling ratio, the same target demonstrations, and comparable adapter capacity and training budget? Please report success under both the original and new conditions.
  • What does intermediate-state prediction contribute to adaptation performance? How does performance change when the transition teacher is trained with action-relation alignment alone, without intermediate-state prediction?
  • How consistent are the improvements across the three sampled target support sets? Please report variability for the main comparisons and component ablations.
Reviewer_Zh9YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates few-shot adaptation of flow-matching Vision-Language-Action (VLA) models (specifically π0.5\pi_{0.5}) under scene shifts (such as camera angles, lighting, background changes, and sensor noise) that leave underlying manipulation semantics unchanged. To bridge the gap between shifted visual observations and the policy's existing control representations, the authors propose ShiftVLA. The framework operates in three main stages:

  • Transition Teacher: A frozen V-JEPA 2-based encoder predicts intermediate segment frames LmidL_{mid} and aligns its neighborhood geometry with action-chunk distances (Lrel\mathcal{L}_{rel}).

  • Control-Grounding: A lightweight reader maps Action-Expert control states into the transition space, jointly fine-tuning the model on source data (Lct\mathcal{L}_{ct}).

  • Correspondence-Guided Adaptation: During target-scene adaptation, low-rank (LoRA) adapters are modulated by source-prototype coordinates, and a gradient projection removes action updates that conflict with the established correspondence (Lcorr\mathcal{L}_{corr}).

Critical strengths

I found the following novelties:

  • Methodological Novelty and Transparency: Combines dynamic-aware source representation spaces with correspondence-constrained target parameter updates in a novel way. Additionally, the authors transparently report negative/nuanced findings in the appendix (e.g., partial control-transition agreement after controlling for action similarity).

  • Consistent Empirical Performance: Demonstrates robust improvements across multiple benchmarks, outperforming strong baselines by 1.8 points on LIBERO-Plus, 3.5 points on RoboTwin 2.0 (winning across 17–18 of 20 tasks), and 7.0 points on real-world multi-stage robotic tasks.

  • Valuable Real-World Validation: Conducts extensive real-world robot experiments (5 long-horizon tasks ×\times 4 shifts ×\times 25 trials) providing checkpoint-level progress data that highlight practical utility.

Critical weaknesses

Author should addres the following concerns:

  • Causality Gap (“Better, but not why”): The paper claims success stems from maintaining alignment with the transition teacher, but lacks validation (e.g., tracking Lcorr\mathcal{L}_{corr} during standard LoRA fine-tuning to verify if rising loss correlates with failure). Furthermore, gains may partly stem from extra pre-training compute/V-JEPA 2 feature infusion rather than the transition design itself.

  • Confounded Transition Representation: The transition geometry (Lrel\mathcal{L}_{rel}) is explicitly regularized by action-chunk similarity. Appendix D.2 reveals that after controlling for action similarity, the Action Expert no longer agrees with the teacher more than the VLM does, suggesting the method primarily stabilizes action structures rather than tracking environmental scene shifts.

  • Unquantified Overhead & Missing Baselines: The method introduces significant runtime complexity (querying a transformer reader and distance calculations against KK prototypes at every Action-Expert block and denoising step), yet omits parameter counts and latency evaluations. Real-world comparisons also omit top simulation baselines (FLA, RETAIN).

Other points:

  • There are components of ShiftVLA that have substantial conceptual overlap with existing techniques, and the manuscript does not sufficiently establish what is fundamentally new beyond their integration. For instance, ATI-VLA aligns predictive observation/action latents and adaptively injects them into action generation. Similarly, the Equation 13 proposes modifying the action gradient when it conflicts with the correspondence gradient; however, This is essentially a one-sided application of the conflicting-gradient projection principle used in PCGrad.
Questions for authors

I have the following questions:

a. Correspondence Degradation: Does standard LoRA fine-tuning actually damage control-transition correspondence? Does tracking Lcorr\mathcal{L}_{corr} during training show that runs with higher loss spikes systematically yield lower target success?

b. Ablation of Extra Compute: How much performance gain is attributable to the extra training steps and V-JEPA 2 feature injection, as opposed to the specific transition-referenced design? (e.g., comparing against a baseline trained with Lflow\mathcal{L}_{flow} alone for the same extra duration).

c. Control-Transition vs. Action Similarity: Given that controlling for action similarity removes the distinct correlation advantage of the Action Expert over the VLM (Appendix D.2), can the authors clarify whether the mechanism acts primarily as a regularizer for action structures rather than learning scene-independent dynamics?

d. Inference Latency: What is the precise parameter overhead, memory footprint, and wall-clock latency per action chunk of ShiftVLA compared to standard LoRA-adapted π0.5\pi_{0.5} on target hardware (e.g., RTX 5090)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.