acceptodds
Under review as a conference paper at ICLR 2027

EquiKey: Object-Centric SE(3)-Equivariant Keypose Guidance for One-Shot VLA Adaptation

Abstract

Vision-Language-Action (VLA) models acquire broad semantic and visuomotor priors from large-scale robot data, yet adapting them to novel manipulation tasks remains challenging when only a single demonstration is available. In particular, a generalist VLA may recognize task-relevant objects and reuse familiar motor skills while still failing to infer the precise three-dimensional configurations required for successful execution. We introduce EquiKey, an object-centric SE(3)-equivariant keypose guidance framework for one-shot VLA adaptation. Given a single demonstration of a novel task, a lightweight task-specific SE(3)-equivariant predictor learns to infer a sparse sequence of task-relevant keyposes. These predicted keyposes provide explicit geometric guidance to a pretrained keypose-conditioned EquiKey-VLA, whose backbone remains frozen during task adaptation. During execution, the predicted keypose sequence is temporally scheduled as high-level geometric targets, while EquiKey-VLA generates dense continuous actions conditioned on the latest observation. On 10 one-shot OOD tasks in RoboTwin 2.0, our method achieves average success rates of 77.1% and 76.9% under the Easy and Hard settings, improving the strongest VLA baseline by 42.5% and 44.6% and the strongest task-specific one-shot adaptation baseline by 26.5% and 33.6%, respectively. In real-world one-shot Task-OOD experiments, our method achieves a 73.3% success rate, substantially outperforming the strongest baseline at 11.7%. These results demonstrate that sparse SE(3)-equivariant keypose guidance provides an effective interface for combining geometric inductive biases with the semantic and visuomotor capabilities of pretrained VLA models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, One-Shot Adaptation, SE(3)-Equivariance, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_djAJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes a VLA conditioned on key poses. It extracts key poses from expert demonstrations. Then, it conditions a VLA on these keypose sequences as a form of task description. This allows for one-shot transfer, since a main source of information in the expert trajectory is the macro sequence of actions (key poses) rather than low level controls.They run experiments both in simulation and on hardware showing the effectiveness of their approach.

Critical strengths

SE3 equivariance is a nice property to exploit for robotic systems. Most VLAs just use brute force learning from data, whereas taking advantage of equivariances could improve data efficiency. While SE3 equivariance has been studied before in general, applying it to VLA is still useful.

I like the idea of object-centric conditioning and key poses sequences, which is basically a goal-conditioned method. This is a nice alternative to just using text and images.

Experiments seem thorough.

Critical weaknesses

This paper is somehow very hard for me to follow. I think part of the problem is using EquiKey and EquiKey-VLA as names of the two different components, as these are very similar sounding. The writing also seems to lack an overall structure that motivates and combines the different ideas, it reads like a bunch of disparate ideas connected for this one paper, even though I dont think that is accurate.

I think the equivariance is oversold. Even if Equikey preserves it, the overall framework does not due to learned nature of perception and the VLA. Please correct me if I am missing something.

Questions for authors
  1. Even if Equikey preserves SE3 equivariance, the VLA does not, correct? If true, this somewhat breaks the logic of the paper because SE3 equivariance in terms of actions would not be preserved. Of course you can still use data augmentation, but this does not guarantee equivariance either.
  2. Even if some of the internal processing is equivariant, is the perception as well? Maybe run some ablations on the same scene with SE3 transformations, and see what the error of the corresponding poses are. Also check the action outputs to see if the actions have corresponding changes or not.
  3. How well does this handle clutter? Object-centric methods may be especially susceptible to clutter because distinguishing between objects and their boundaries becomes difficult
  4. Why do you have to retrain the key pose predictor on the expert trajectory? This would be much stronger if key pose estimator was transferable to new scenes.
  5. If I understand correctly, you finetuned the pi0.5 baseline on the same 40 tasks as used to train Equikey-vla. Correct?
Reviewer_sFMbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a hierarchical framework, called EquiKey, for adapting a VLA policy to new tasks using one single demonstration. To prepare the training data, a VLM is used to segment the demonstration and SAM3 is used to provide object-centric supervision. A equivariant light-weight predictor is trained to produce a plan of end-effector poses from the initial RGB-D observation. These poses then condition a pre-trained VLA policy to predict the final actions to execute. The method demonstrate strong one-shot performance on new tasks in both simulation and real-world tasks.

Critical strengths
  1. The motivation and the explanation of the whole method is clear.
  2. The performance of the proposed method in both simulation and real-world are promising.
Critical weaknesses
  1. None of the compared baseline methods leveraging SE(3)-Equivariance. The experiments does not well justify the performance gain from equivariance or from hierarchical architecture using keypose as the bridge.
  2. The fixed scheduler might lead to unrobustness. According to section 3.4, there is not a specific criterion for judging if current phase is finished and if we can move to next planned keypose. A successful execution requires that the low-level Equikey-VLA execute the actions to reach the desired keypose within the time (defined by rir_i) without mistake. There might be situation where, under different task configuration, such as different initial and goal pose, the policy needs different number of time steps for finishing one phase.
Questions for authors

Clarification request

  1. To adapt existing VLA to Equikey-VLA, do the parameters from original VLA (such as the VLM) need to be finetuned? If so, please provide more training details. Are they trained with 40 seen tasks?
  2. Does SE(3)-equivariance ensure the generalization to new task configurations? Does the statement require that a successful trajectory under new scene can be obtained by transforming the source demonstration trajectory?
  3. Does b_i in equation (1) used later?

Suggestion

  1. The policy performance of using one demonstration for one-shot experiment will highly depend on this single demonstration. Reporting the mean and standard deviation of the results across diffferent chosen demo and seed will be more convincing.
  2. Please add citations to acknowledge SOTA methods SAM2ACT[1], BridgeVLA[2], CLAP[3] in "Keypose Guidance and Hierarchical Manipulation" of Related Work section.

[1] SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation. Fang et al. (ICML 2025).

[2] BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models. Li et al. (NeurIPS 2025).

[3] Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints. Hu et al. (ICLR 2026).

Reviewer_cYDbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EquiKey, a framework for adapting a pretrained VLA to a new manipulation task using one demonstration. A VLM identifies milestone timestamps, acting arms, and relevant objects in demonstration videos. End-effector poses at those timestamps become keypose targets, while segmentation provides object-localization supervision. A lightweight, task-specific SE(3)-equivariant predictor generates an ordered keypose sequence from the initial RGB-D observation. A shared keypose-conditioned VLA then produces continuous actions using current observations and time-scheduled guidance. Only the predictor is trained during task adaptation; the VLA remains frozen.

The method achieves 77.1%/76.9% success on ten held-out RoboTwin tasks under Easy/Hard settings, compared with 50.6%/43.3% for the strongest reported adaptation baseline. It also reports 73.3% success on three real-world held-out tasks. The results support the usefulness of geometric guidance for one-shot adaptation, although the contribution of equivariance and the scope of object-pose generalization need further clarification.

Critical strengths
  • The architecture addresses a practical problem. Learning task-specific geometric targets separately from continuous action generation is a sensible way to reuse pretrained control priors. The common end-effector pose representation provides a straightforward interface between different tasks and the shared VLA.

  • The reported improvements are substantial. The gains extend across simulation and real-world tasks. Several multi-object tasks also perform well, including Pick Dual Bottles and Place Cans Plasticbox. These results provide meaningful evidence that the overall system can handle more than isolated single-object manipulation.

  • The ablations help explain the execution interface. The paper examines conditioning location, modality dropout, scheduling, and direct interpolation. The drop from 77.1%/76.9% to 55.6%/53.5% with interpolation supports the value of observation-conditioned VLA execution beyond simply following sparse poses.

  • The implementation is described in useful detail. The appendix includes architecture, losses, adaptation time, and a conditional equivariance analysis. The authors also acknowledge limitations of initial-observation-only prediction and sparse keypose guidance.

Critical weaknesses
  1. The experiments do not isolate the benefit of equivariant prediction. Table 3 varies conditioning, dropout, scheduling, and execution, but does not compare equivariant and non-equivariant predictors under matched supervision and augmentation. The gains could therefore come largely from supplying explicit geometric targets. Keeping the VLA and scheduler fixed while changing only the predictor would directly test the main claim. A stronger keypose-based alternative would also help establish the value of the proposed architecture beyond the interpolation baseline.

  2. The evidence for independent object-pose generalization needs clarification. Appendix C explicitly states the assumptions behind its whole-cloud equivariance result; I do not view this as an error in the proof. The remaining question is how the evaluated system handles objects moving independently against a fixed background. Section 3.1 explains stage grouping, but does not show which objects anchor grasp and placement keyposes in an actual relational task. Since the local pose model uses only its crop and stage index (Appendix B.2), these assignments matter. Clarifying the source/destination pose randomization in the existing evaluations would help establish the scope of the results.

  3. The real-world comparison omits the stronger adaptation baselines. Tables 2 and 10 compare against pooled-data VLA baselines, while the task-specific LoRA, FLA, and FTM baselines appear only in simulation. The real-world gain is encouraging, but comparison with the strongest simulation adaptation baseline would better support the claimed advantage of this adaptation strategy.

Questions for authors
  1. For one existing pick-and-place task, can the authors show the object assigned to each keypose and its local crop? Are source and destination poses randomized independently during evaluation?

  2. Section 4.3 treats oracle guidance as an upper bound, but Table 8 reports 95% success with predicted keyposes versus 41% with oracle keyposes on Place Cans Plasticbox (Hard). How are scene-specific oracle targets obtained, and what explains this reversal? This needs clarification before interpreting the oracle gap as evidence of prediction quality.

  3. Is there evidence on sensitivity to the chosen demonstration or on the accuracy of the VLM's milestone and object assignments, beyond agreement with the resulting pose labels?

Comments on the paper

1 comment
Area_Chair_68xPOpenReview ↗

Rebuttal Discussion

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.