- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the problem of finetuning VLAs under scene shifts. Instead of directly finetuning to data collected under the new scene, the paper proposes to learn an invariant representation of the robot's state change, called "execution transitions." They then establish a correspondence between the "execution transtions" and the latent representations in the VLA's Action Expert. This correspondence is then used to guide the finetuning of the Action Expert. The finetuning involves encoding a few "prototype" source domain trajectories into a coordinate, which is used to guide the LoRA weight updates to the target domain. The paper evaluates on two simulation benchmarks, LIBERO-Plus and RoboTwin, each grouping tasks into source and target domains. The proposed ShiftVLA achieves the highest overall performance across both benchmarks in the few-shot setting. The paper further evaluates on a real GALAXEA robot platform, showing more efficient finetuning under scene shifts compared to OpenVLA and Pi-0.5.
Critical strengths
- The paper targets the imporant problem of efficient finetuning across domain shifts.
- The real robot experiments seem impressive.
Critical weaknesses
- The method is unnecessarily complex. They first establish a correspondence between the robot's state change and the VLA action expert's hidden states. The state change, or "execution transition," is captured by an encoder that takes in the initial and final image observations of a trajectory segment, . It is trained to (1) reconstruct intermediate imaged features conditioned on some progress , (2) keep the distance between the s closely distributed to the distance between the actions . (this doesn't quite make sense, as they defined a distribution proportional to distance and then minimize their KL divergence, see Eqn 5, 6). Then, having trained this delta state encoder, they train another mapping from the VLA's action expert's latent states to the delta state space. During finetuning, they first construct a "transition-reference coordinate" from a couple of prototype transitions from the source domain. This coordinate is then injected to the LoRA finetuning for guidance. Finally, the delta state representation is also used to shape the gradient update so that only correspondence-preserving gradient directions are kept.
- My high-level understanding of the algorithm is: train a joint representation space where the state change and the corresponding action chunk are mapped close to each other. Then finetunine the VLA action expert while preserving this correspondence.
- The ablations in Table 4 only covers a small subset of the algorithm's design choices. There are between 10-20 design choices, most of which seem arbitrary.
- Following from the first point, the writing is extremely terse and borderline incoherent. The entire paper is littered with undercontextualized jargons. A few examples:
- Lines 47-49: "We instantiate this view in SHIFTVLA, which learns a transition reference, establishes its correspondence with Action-Expert control states, and organizes few-shot scene adaptation around the resulting control–transition correspondence." The paper does not provide enough context for the reader to understand what "transition reference," "Action-Expert control states," or "control-transition correspondece" mean.
- Lines 75-77: "Source-calibrated control readouts modulate low-rank control refinement across Action-Expert depth, while gradient projection removes action-update components that would weaken the grounded correspondence." This is extremely confusing.
- Lines 166-167: "We denote by the distance between layer-normalized features." is completely out of place. This is not used until several lines later, and when I got there it felt like was never introduced.
- This can also be seen from Figure 2, which is extremely dense and hard to understand.
- Overall, this paper needs signficant overhaul to reach a state that is ready for peer-review. This is beyond the scope of a rebuttal.
Questions for authors
- Can you provide a brief description of each baseline being compared to, e.g. FLA, Robotic Sttering, and RETAIN?
- Can you compare to efficient finetuning baselines on the real robot? The simulation results show slight improvement over effcient-finetuning baselines like FLA and RETAIN. It's unclear if this trend holds on a real robot.
- Can you provide a more comprehensive ablation of the algorithms individual design choices? E.g. the use of KL divergence for ?