acceptodds
Under review as a conference paper at ICLR 2027

Learning Latent Atomic Movements for Generalizable and Adaptable Robot Policies

Abstract

Adapting robot policies from limited demonstrations is challenging when task-dependent movement selection and precise control are learned jointly. We propose an atomic-movement interface that enables adaptation through movement selection while keeping the policy fixed. A latent action model learns compact movement representations as sparse combinations of shared codebook entries, supervised by action reconstruction. A latent prior predicts these representations from the instruction and observation history, and an observation-conditioned policy generates the corresponding actions. To encourage the policy to follow the movement condition, we introduce Latent Override, which pairs an observation with a compatible donor movement and its action target during training. After learning this interface, adaptation updates only the lightweight prior adapter while keeping the movement representation and policy fixed. Using Cosmos-Policy as the policy backbone, our method achieves 65.5% average success across ten real-world base tasks. Under few-shot adaption setting, it achieves 58.33% average success across six adaptation tasks, compared with 50.0% for Cosmos-Policy and 55.0% for under full-model fine-tuning, with lower measured adaptation time and GPU requirements. These results support reusing a learned movement interface and a fixed policy for few-shot robot policy adaptation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r2nwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper puts a chunk-level latent between language and a frozen Cosmos-Policy controller. A latent action model encodes a demonstrated video–action chunk as M tokens, each defined as a sparse convex combination of a shared codebook (Eq. 1), and is trained by action reconstruction (Eq. 5). A Qwen3.5-0.8B prior predicts those tokens from the instruction and a short observation history; the world–action model then generates the 16-step action chunk from the current observation and the latent, with its own text condition removed. Latent Override keeps an observation and substitutes a donor latent and action target (Eq. 6). After this interface is learned, a new task updates only the prior side. On one Franka, the paper reports 65.5% average success on ten base tasks (Table 1) and 58.33% on six few-shot tasks with 6–8 demonstrations (Table 2), at 97 minutes on one GPU (Table 3).

Critical strengths

Freezing the controller is carried through on a real robot, and in-distribution success holds. Few-shot adaptation updates the prior side while the LAM, codebook, and world–action model stay fixed (Section 3.3, Figure 2). Table 3 measures that choice at 97 minutes, one GPU, and 8.2 GB, against 200 and 124 minutes on four GPUs and 80 GB and 93 GB for full-model fine-tuning of Cosmos-Policy and π0.5. This credits a narrower fact: with the world–action model held fixed, base-task success stays at 65.5%, beside π0.5 at 66.0% (Table 1). The lower adaptation cost in Table 3 is a property of updating a small prior head. Whether that cost or the few-shot average identifies the atomic-movement interface is the subject of the second weakness.

The predicted movement is visible on real executions, and Latent Override moves the controller toward it. Figure 4 decodes the prior’s latent through the LAM action decoder and shows the resulting arrow beside the arm’s subsequent frames. Table 4a reports the L1 between that decoding and the world–action model’s output as 0.1087 with Latent Override and 0.1535 without it, measured during real-world execution (Section 4.4). These results speak to agreement on the logged and illustrated rollouts.

Critical weaknesses

The few-shot average does not identify a generalization effect. The adaptation claim is the 58.33% versus 50.0% and 55.0% in Table 2. The margins are 8.3 and 3.3 points, with no trial count and no interval. The difference is also uneven across the six tasks. Cosmos-Policy leads novel-object T2 by 75 to 55. π0.5 matches or leads both long-horizon tasks, 70/70 versus 70/65. The larger gaps are in novel arrangements, 60/40 versus 40/20 and 45/25, and Table 6 gives those two tasks the same instructions as base T1 and T4 in Table 5. Section 5 states that the adaptation tasks share movement components with the base tasks. Table 2 therefore records a small, uneven difference on reused skills, concentrated on two training instructions in a new layout.

The novel piece is not what the success tables measure. Section 2.2 locates the difference from prior latent-skill work in two places: quantized skills already support few-shot transfer with a frozen decoder, and villa-X already factorizes latent planning from action generation. The manuscript’s own distinction is the shared movement code plus Latent Override, which is meant to stop an observation-conditioned world–action model from ignoring its latent (Section 2.2). Table 4a is the only measurement of that distinction, and it is an action L1 of 0.1087 versus 0.1535, not task success. Tables 1–3 instead compare a small prior head, with the 2B world–action model frozen, against full-model fine-tuning of Cosmos-Policy and π0.5 for 500 steps (Section 4.1, Appendix C). A small module trained for the same step budget will also be cheaper (Table 3: 8.2 GB on one GPU versus 80 GB and 93 GB on four) and can match success without the codebook being the cause. There is no control that keeps this frozen world–action model and replaces the atomic-movement prior with a similarly small language-conditioned adapter, and no comparison to the frozen-decoder or latent-planning baselines cited in Section 2.2. The reported adaptation result is therefore what parameter-efficient use of a strong backbone would be expected to show. It does not establish a new movement interface.

Questions for authors

Under the Table 2 budget, freeze this same world–action model and adapt a language-conditioned module with the same number of trainable parameters, predicting actions with no atomic-movement codebook. Report per-task success next to Table 2. If that module matches 58.33%, the adaptation result does not depend on the movement interface.

3 comments
AuthorsOpenReview ↗

Response to Reviewer r2nw

We thank the reviewer for the constructive feedbacks and suggestions. We are glad that the reviewer finds our latent override design meaningful, and acknowledges our real-world experiments. We will then address the reviewer's remaining concerns.

W1.

Our evaluation tests primarily whether learned movements can support new objects, layouts, and task sequences with only 6–8 demonstrations and a frozen execution policy. Reusing movement components is central to this goal. Keeping instructions unchanged while changing layouts isolates spatial generalization: the same goal requires a different trajectory. On these novel arrangement tasks, our average success is 50%, versus 30% for Cosmos-Policy and 35% for π0.5π_{0.5}. Across all six tasks, ours achieves the highest observed average while updating only the prior adapter. We do not claim a uniform advantage, yet the results show competitive adaptation with substantially lower training cost. Each task uses 20 trials.

W2/Q1.

We thank the reviewer for the valuable feedback. First, we further conduct an experiment measuring our latent override design's actual impact on real-robot experiments:

MethodT1T2T3T4T5T6T7T8T9T10Avg
Cosmos-Policy Baseline4555802530556585406554.5
Cosmos-Policy w/o latent override4550803030456085455052.0
Ours7065905565505595605065.5

The results show that, without latent override, success rate is comparable to or below Cosmos-Policy baseline on all base tasks. In the deloyment we observe that the policy trained without latent override is 'reciting trajectories' seen during training stage purely based on visual observation, regardless of what the latent prior outputs. This verifies that the latent override design is vital to train a usable atomic movement-conditioned policy.

Second, following the reviewer's suggestion on comparing against a language-conditioned adapter, we add Cosmos-Policy + LoRA (using rank 8, same as our latent prior) under the same few-shot protocol (the same few-shot demonstrations and 500 adaptation steps ) and explicitly compare the compute resources used.

MethodT1T2T3T4T5T6Avg
Ours60556040706558.33
Cosmos-Policy + LoRA55702025455044.17

The result shows that with LoRA finetuning, Cosmos-Policy performs worse than baseline, because model has to learn by itself from the very limited few-shot demonstrations. Given the novelity of the new tasks, this learning process could be quite difficult. Our method, on the other hand, solves this problem through a highly composable vocabulary, boosting the learning of new tasks.

We also compare the efficiency of different methods:

MethodTraining time per taskGPUsPeak memory per GPU
Ours9718.2
Cosmos-Policy, full tuning200480
Cosmos-Policy + LoRA140420

The results show that the LoRA version still costs significantly more expensive than our method, primarily due to the fact that the full WAM model still needs to be loaded and forwarded, while our method only requires loading and updating a much smaller model.

Dear Reviewer r2nw, thank you again for your time and thoughtful feedback. We hope that our additional clarification and experiments have addressed the concerns and questions raised in your review. We would greatly appreciate it if you could let us know whether your concerns have been addressed. If you have further questions, we would be very happy to discuss.

Thanks the authors for the detailed reply. My concerns are largely addressed and I am happy to raise my score.

Reviewer_EK4WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the paper introduces an atomic-movement-based framework for robot policy learning and few-shot adaptation. The method learns a codebook to represent short action sequences and uses a latent prior to predict these movements from language instructions and observation. A latent-conditioned policy then generates robot actions, while the proposed Latent Override mechanism encourages the policy to follow the movement conditions. When there is new tasks come, only the prior adapter is updated while keeping the policy frozen.

Critical strengths
  1. I think the overall idea of this paper is meaningful, especially for fast adaptation to new robot tasks. Instead of fine-tuning the entire policy, the authors try to reuse learned movements and only adapt the movement selection process. I think this is a practical direction and could be useful for real-world robot deployment.

  2. Experiments on real robots achieves competitive performance while requiring much less GPU memory and computational resources for adaptation. The efficiency advantage is valuable.

  3. I find the Latent Override mechanism an interesting part of this work. The authors recognize that the policy may ignore the latent movement condition and rely mainly on visual observations. The proposed training strategy tries to address this problem directly, which I think is a reasonable design and potentially useful for other latent-conditioned robot policies.

Critical weaknesses
  1. My main concern is that several important ideas in this paper have already been explored in previous studies. For example, [1] investigates efficient adaptation through latent representations while keeping the foundation policy fixed, [2] also separates latent action prediction from low-level robot control, and [3] learns reusable skill representations using a quantized codebook. I think the authors should explain what is fundamentally different in their framework, rather than (sounds like) combining these existing designs.

  2. One important claim of this paper is that the learned codebook captures reusable and compositional movements. However, I am not fully convinced by the current results. For example, Figure 3 does not show very clear separation between different movement properties, and I wonder whether similar patterns could also be observed with a standard action autoencoder. The main paper describes sparse linear combinations of codebook entries, but Appendix A.2 appears to use hard code selection during the forward pass. This makes it unclear where the claimed compositionality actually comes from. I think more direct experiments are needed to demonstrate that the learned codes provide meaningful advantages for generalization.

  3. I think the current experiments do not fully explain where the improvements come from. For Latent Override, Table 4 only reports the L1 distance between the reconstructed and predicted actions, rather than showing whether it improves actual task success. Also, the donor selection is mainly based on the robot's initial configuration, which may not guarantee that the transferred actions are compatible with the current object arrangement. Another concern is that the authors compare their lightweight adaptation mainly against full-model fine-tuning baselines. I would expect comparisons with LoRA or other parameter-efficient adaptation methods to better justify the claimed efficiency advantage. These missing evaluations make it difficult to isolate the actual contribution of each proposed component.

Questions for authors
  1. Could the authors clarify the main methodological differences between the proposed framework and prior works [1], [2], and [3], particularly regarding latent movement representations and frozen-policy adaptation?

  2. The main paper describes sparse linear combinations of codebook entries, but Appendix A.2 appears to use hard selection during the forward pass. Could the authors clarify how compositionality is achieved in the actual implementation? Is there any comparison with standard discrete codebooks or soft code combinations?

  3. Could the authors provide downstream task success rates with and without Latent Override? Also, how is the compatibility between donor actions and retained observations ensured when object configurations differ?

  4. I wonder if the authors compare against parameter-efficient fine-tuning baselines, such as Cosmos-Policy with LoRA or lightweight adapters, to better isolate the benefits of the proposed framework?

  5. Figure 3 does not show very clear separation of different motion properties. Could the authors provide quantitative evidence that the learned codes capture consistent and reusable movement semantics across tasks?

[1] Da et al. Latent Adaptation of Foundation Policies for Sim-to-Real Transfer. ICLR 2026. [2] Chen et al. Villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models. [3] Mete et al. Quest: Self-supervised skill abstractions for learning continuous control. NeurIPS 2024.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EK4W

We than the reviewer for the detailed and valuable suggestions and for recognizing our efficiency and latent override design. We will address the remaining concerns as follows.

W1/Q1.

Thank you for your mentioning of several related works. Compared with previous literatures, our method differs in what latents represent, and what adaptation process changes:

(1) Found-adapt solves sim-to-real transfer problem, focusing on filling the dynamics gap between simulator and real-world, which does not discuss policy adaption on novel movements and task scenarios. We adapt movement selection to new tasks using demonstrations, language, and observations, which is another research focus yet carries different challenges.

(2) Villa-X primiraly focuses on visual latents and jointly models latent and robot actions. However visual latents sometimes cannot reflect very accurate spatial movements, therefore our method decides to learn more expressive latents coming from raw trajectories. And different from training a full model, we focuses on efficient adaption by training a general WAM + a light-weight decoder to respond to movement conditions.

(3) QueST learns quantized action sequences and supports frozen-decoder transfer. However, their design is inherently different to ours. Instead of learning the 'instructions' as we do, they train the model to output these skill tokens, which are then decoded into actual trajectories. The performance is dependent on self-designed transformer and skill decoder, and it can't reuse the current SoTA VLA/WAM methods. Our method, on the other hand, treats action latents as the same as language instruction, therefore could be easily integrated into any SoTA policy models.

From a higher level perspective, our method's design phylosophy is to replace the natural language instructions with trajectory-supervised atomic movement instructions. In this way we lift parts of generalization burden from policy to the latent level, thus achieving faster adaptation and better generalization ability. This motivation and pipeline is novel and distinct from all the three works above. What's more, another core technical contribution in our work is latent override, which trains the policy to distinguish alternative movements in similar contexts. We will add discussions with all these mentioned papers in the related work section.

W2/Q2/Q5.

We clarify that in our actual experiments, the implemented design uses six tokens and a shared codebook of 32 vectors. Each token selects two entries and computes zm=wm1ci+wm2cjz_m = w_{m_1} c_i + w_{m_2} c_j, with nonnegative weights summing to one. The six mixed tokens jointly describe an action chunk. We thank the reviewer for pointing out the confusion and we will fix the expression in the Appendix that may cause misunderstandings.

Since in the base tasks we collected, most trajectories involve complecated movements, spanning directions on several axes. Therefore separation on translation (Δx,Δy,Δz\Delta x, \Delta y, \Delta z) might not be that significantly obvious. In order to better prove that our learned atomic action codebook actually captures semantics across tasks, we conduct an additional experiments: we use our trained LAM to reconstruct any trajectories from the dataset; then we compute the action L1 distance between original trajectory and reconstructed trajectory decoded through learned codes combination. We compare this calculated L1 distance on base tasks and novel tasks as follows:

Trajectory sourceAction L1 ↓\downarrow
Base tasks0.0852
Novel tasks0.0934

The results show that, on novel tasks involving new scenarios and movements, trajectories reconstructed from our learned codes still achieve performance comparable to that on the base tasks. This demonstrates that the learned codebook transfers effectively and generalizes to new action semantics and trajectories.

AuthorsOpenReview ↗

Response to Reviewer EK4W (continual)

W3/Q3/Q4.

(i) Latent override's actual impact on real-robot experiments

We carried out experiments on the base tasks:

MethodT1T2T3T4T5T6T7T8T9T10Avg
Cosmos-Policy Baseline4555802530556585406554.5
Cosmos-Policy w/o latent override4550803030456085455052.0
Ours7065905565505595605065.5

The results show that, without latent override, success rate is comparable to or below Cosmos-Policy baseline on all base tasks. In the deloyment we observe that the policy trained without latent override is 'reciting trajectories' seen during training stage purely based on visual observation, regardless of what the latent prior outputs. We hope this real-world experiment result can further suppot our claim that the latent override design is vital to train a usable atomic movement-conditioned policy.

(ii) Clarification on donor selection

The goal of latent override is to encourage the policy to follow the movement condition while retaining observation-conditioned execution through ordinary demonstrations. Pose/gripper matching provides approximate kinematic compatibility, and help the model learn to react regardless of current object pose/contact status. Besides, latent override setting only occupies 10% of training examples, and uses action-only supervision with loss weight 0.1 to limit its influence. Therefore the original manipulation ability inherited in the policy is not lost during the training.

Additional object/contact matching could improve scene compatibility, but there are two potential problems when doing so: (i) Adding object pose matching might create another uncertainty factor stemming from the pose detection errors, which might not be as accurate as proprioception state. (ii) If adding object pose matching, there is a much higher possibility that the donor is from the same scenario, thus their action is almost identical to the current sample. And in this case the latent override would be meaningless in making world model aware of different condition actions.

(iii) Comparison with LoRA

We conduct additional experiments on Cosmos-Policy with LoRA finetuning. Please refer to our response to W2 of reviewer 7PDy for detailed results, efficiency data and analysis.

Thank authors for further explanation, I would encourage adding a comprehensive discussion on broader techniques paths in Q1. and include the extra experiments in an updated version of paper. My concerns are partially addressed and I'd like to raise my initial evaluation to 3.

Reviewer_7PDyOpenReview ↗
Rating
2(−1)
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose atomic movement interface for adapting robot policies from few demos. A latent action model encodes action chunks as sparse combinations over a shared codebook, a prior predicts these codes from language and observation history, and a frozen WAM does the actual execution. The paepr also propoase Latent Override as a training mechanism, meant to force the WAM to actually listen to the latent. Across six real-world adaptation tasks they get 58.33% average success while only updating the prior adapter.

Critical strengths
  1. The basic idea of splitting “which movement to do” from “how to do it in this scene” is interesting. It’s well motivated for low-data adaptation.
  2. Latent Override targets a real failure mode, where the policy just ignores the latent and leans on observations. The latent-following L1 dropping from 0.1535 to 0.1087 is a reasonable sign that it works.
  3. The real-robot experiments span several kinds of manipulation, and I appreciate that adaptation time and memory are reported next to success rates.
Critical weaknesses
  1. The claims of generalization claims are not well supported convincingly. Everything runs on one robot, and the six adaptation tasks reuse movement components from the base tasks, so it’s unclear how far this goes beyond that.
  2. The baseline comparison is hard to read. The method trains only a prior adapter, whereas the baselines are fully fine-tuned, and the proposed method uses one GPU against four for the baselines.
  3. The validity of Latent Override is worth concern. Donors are matched on robot pose and gripper state only. According to the appendix, object pose, contact state and the actual movement difference aren’t checked, so a swapped latent may not even make sense for the retained scene.
Questions for authors
  1. How many trials per task, and how many independent training runs, are behind each success rate? Could you add uncertainty intervals for the few-shot comparison?
  2. What fraction of attempted overrides are accepted? And does performance change if donor matching also accounts for object configuration or contact state?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7PDy

We thank the reviewer for the constructive feedback, as well as recognition of our method design and real-robot evaluation. We will address reviewer's remaining concerns as follows:

W1.

Our experiments evaluate compositional generalization: adapting learned movements to new objects, spatial arrangements, and longer task sequences. The base tasks cover stacking, pick-and-place, pushing, and tool use; the six adaptation tasks introduce distinct changes beyond these training tasks. Reusing movement components is central to this evaluation and the question is whether they can support new tasks through prior adaptation while keeping the execution policy fixed. With only 6–8 demonstrations per task, our method achieves 58.33% average success, supporting this form of generalization. A broader and more diverse base movement set offers a natural path toward expanding the tasks to more general scenarios.

Our physical evaluation uses the available Franka platform. However the principle of representing and composing movements through a latent interface is not specific to any robot, making extension to other embodiments a promising direction.

W2.

We thank the reviewer for raising the question. Updating only the prior adapter is a central benefit of our design: the learned execution policy can be reused during adaptation. To directly compare lightweight adaptation strategies, we add Cosmos-Policy + LoRA (using rank 8) under the same few-shot protocol (the same few-shot demonstrations and 500 adaptation steps ) and explicitly compare the compute resources used.

MethodT1T2T3T4T5T6Avg
Ours60556040706558.33
Cosmos-Policy + LoRA55702025455044.17

The result shows that with LoRA finetuning, Cosmos-Policy performs worse than baseline, because model has to learn by itself from the very limited few-shot demonstrations. Given the novelity of the new tasks, this learning process could be quite difficult. Our method, on the other hand, solves this problem through a highly composable vocabulary, boosting the learning of new tasks.

We also compare the efficiency of different methods. We use 4 GPUs to finetune π0.5\pi_{0.5}, Cosmos-Policy and its LoRA version primarily to align the training paradigm of their original baseline. And single GPU training for merely 500 steps are usually not enough for these models.

MethodTraining time per taskGPUsPeak memory per GPU
Ours9718.2
Cosmos-Policy, full tuning200480
Cosmos-Policy + LoRA140420

The results show that the LoRA version still costs significantly more expensive than our method, primarily due to the fact that the full WAM model still needs to be loaded and forwarded, while our method only requires loading and updating a much smaller model.

W3/Q2.

The goal of latent override is to encourage the policy to follow the movement condition while retaining observation-conditioned execution through ordinary demonstrations. Pose/gripper matching provides approximate kinematic compatibility, and help the model learn to react regardless of current object pose/contact status. Besides, latent override setting only occupies 10% of training examples, and uses action-only supervision with loss weight 0.1 to limit its influence. Therefore the original manipulation ability inherited in the policy is not lost during the training.

Additional object/contact matching could improve scene compatibility, but there are two potential problems when doing so: (i) Adding object pose matching might create another uncertainty factor stemming from the pose detection errors, which might not be as accurate as proprioception state. (ii) If adding object pose matching, there is a much higher possibility that the donor is from the same scenario, thus their action is almost identical to the current sample. And in this case the latent override would be meaningless in making world model aware of different condition actions.

Q1.

Each few-shot per-task rate uses 20 robot trials and one training run per setting. We provide 95% Wilson intervals per task and bootstrap intervals for the six-task mean, resampling trials within tasks. Our mean is 58.33% [50.00%, 66.67%]. These intervals measure evaluation uncertainty for each trained policy. Due to the limited time, we can't afford carrying out a few more full pipeline training runs, but we will add the variability across training runs in the final manuscript.

While the authors frame adaptation as compositional generalization, reusing base primitives on minor task variants is merely interpolation, and achieving only 58.33 success on a single robot fails to support true generalization. The claim is still unconvincing I will change my score to 2.

We appreciate the reviewer for the follow-up. Below, we would like to clarify the scope of our generalization claim.

1. Our claim is compositional generalization via few-shot adaptation

We do not claim zero-shot generalization to arbitrary tasks or unseen motor skills. Our setting is few-shot adaptation, that is, given only 6–8 demonstrations of a novel task, whether the robot can acquire it by recomposing previously learned atomic movements, especially with very low computing resources. In our paper, "generalization" refers to this compositional generalization, i.e., solving tasks absent from base training through new combinations of learned movements in new scenes. Its breadth depends on the diversity of the base tasks, so a more diverse base task dataset could yield a richer movement vocabulary and thus a broader space of tasks reachable through composition. Our experiments instantiate this with ten base tasks across four manipulation families.

We thank the reviewer for pointing this out and we will state this scope explicitly in the introduction more clearly.

2. The adaptation tasks are not minor variants.

We would love to clarify that our few-shot adaption tasks are not trivial for current policies:

  • Our Novel objects few-shot tasks require a different manipulation strategy from the base tasks. For example, the lemon is round and tends to roll under the sweeping tool rather than slide like the cubes in the base sweeping tasks, so the sweeping motion for it is much slower and softer, in order to keep it under control. The task does not appear in the WAM's robot training data, and the WAM is not updated during adaptation.
  • Our Novel arrangements few-shot tasks place objects in unseen spatial configurations, so the task requires a completely different trajectory that must be inferred from the scene. If these tasks were interpolations of the base data, a strong policy should transfer with little loss. However, starting from the same base-task average as ours, π0.5\pi_{0.5} loses 27.5 points even after full-model fine-tuning, while our method loses 12.5 points without updating any policy parameters:
Taskπ0.5\pi_{0.5} baseπ0.5\pi_{0.5} few-shotOurs baseOurs few-shot
Stack the blue cup into the grey cup65457060
Pick up the orange cube and place it in the bowl60255540
Average62.535.0 (−27.5)62.550.0 (−12.5)

Regarding the 58.33% average, it is achieved with 6–8 demonstrations and no policy update. Under the same protocol, full-model fine-tuning reaches 50.0% (Cosmos-Policy) and 55.0% (π0.5\pi_{0.5}), all using four GPUs. These success rates reflect the difficulty of real-world few-shot adaptation rather than a weakness specific to our method. Again, we want to emphasize that our contribution is not only improving the adaption performance on novel tasks, but also that we only need far fewer computing time and resources, which make it practical in some light-weight task adaption scenarios.


Finally, we would be grateful to know whether our previous responses have addressed the reviewer's other concerns. If any concern remains, we would be happy to clarify further during the remaining discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

Dear Area Chair and Reviewers,

Thank you for participating in the discussion, and thank you to the reviewers for their careful and constructive evaluation. Reviewers broadly agreed that separating "which movement to perform" from "how to execute it in the current scene" is well motivated for low-data adaptation. They also found that our latent override design useful, and that the real-robot evaluation, together with reported adaptation time and memory, is practical and valuable.

We addressed each reviewer's point/concern with new experiments and clarifications, and we have updated the manuscript accordingly; all revisions are highlighted in red color.

Scope of the generalization claim (7PDy, r2nw): We now state explicitly in the introduction that our claim is compositional generalization through few-shot adaptation: acquiring tasks absent from base training from 6–8 demonstrations by recomposing learned atomic movements, with the policy kept fixed. We do not claim zero-shot transfer to arbitrary tasks or unseen motor skills. We also explain that the novel-arrangement tasks deliberately keep the base instructions, so the same goal must be reached through a different trajectory inferred from the scene. On these tasks, π0.5\pi_{0.5} and our method start from the same base-task average. Yet π0.5 drops by 27.5 points after full fine-tuning, while ours drops by only 12.5 points without updating any policy parameters.

Parameter-efficient baseline and compute (7PDy, EK4W, r2nw): We added Cosmos-Policy + LoRA under the same few-shot protocol. It reaches 44.17% average success, below both full fine-tuning and our method. It still needs 140 minutes on four GPUs, compared with 97 minutes on one GPU for ours. These results are now included in Tables 2 and 3.

Latent Override and task success (EK4W, r2nw): Beyond the latent-following L1, we now report real-robot success on the ten base tasks. Without Latent Override, average success drops from 65.5% to 52.0%, below the Cosmos-Policy baseline. In deployment, this policy tends to replay training trajectories from visual cues regardless of the latent.

Relation to prior work (EK4W, r2nw): We added a discussion comparing our method with Found-adapt, villa-X, and QueST.

We thank the reviewers again for their engagement during the discussion. We believe these revisions make the paper's scope, contributions, and evidence clearer.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.