- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper puts a chunk-level latent between language and a frozen Cosmos-Policy controller. A latent action model encodes a demonstrated video–action chunk as M tokens, each defined as a sparse convex combination of a shared codebook (Eq. 1), and is trained by action reconstruction (Eq. 5). A Qwen3.5-0.8B prior predicts those tokens from the instruction and a short observation history; the world–action model then generates the 16-step action chunk from the current observation and the latent, with its own text condition removed. Latent Override keeps an observation and substitutes a donor latent and action target (Eq. 6). After this interface is learned, a new task updates only the prior side. On one Franka, the paper reports 65.5% average success on ten base tasks (Table 1) and 58.33% on six few-shot tasks with 6–8 demonstrations (Table 2), at 97 minutes on one GPU (Table 3).
Critical strengths
Freezing the controller is carried through on a real robot, and in-distribution success holds. Few-shot adaptation updates the prior side while the LAM, codebook, and world–action model stay fixed (Section 3.3, Figure 2). Table 3 measures that choice at 97 minutes, one GPU, and 8.2 GB, against 200 and 124 minutes on four GPUs and 80 GB and 93 GB for full-model fine-tuning of Cosmos-Policy and π0.5. This credits a narrower fact: with the world–action model held fixed, base-task success stays at 65.5%, beside π0.5 at 66.0% (Table 1). The lower adaptation cost in Table 3 is a property of updating a small prior head. Whether that cost or the few-shot average identifies the atomic-movement interface is the subject of the second weakness.
The predicted movement is visible on real executions, and Latent Override moves the controller toward it. Figure 4 decodes the prior’s latent through the LAM action decoder and shows the resulting arrow beside the arm’s subsequent frames. Table 4a reports the L1 between that decoding and the world–action model’s output as 0.1087 with Latent Override and 0.1535 without it, measured during real-world execution (Section 4.4). These results speak to agreement on the logged and illustrated rollouts.
Critical weaknesses
The few-shot average does not identify a generalization effect. The adaptation claim is the 58.33% versus 50.0% and 55.0% in Table 2. The margins are 8.3 and 3.3 points, with no trial count and no interval. The difference is also uneven across the six tasks. Cosmos-Policy leads novel-object T2 by 75 to 55. π0.5 matches or leads both long-horizon tasks, 70/70 versus 70/65. The larger gaps are in novel arrangements, 60/40 versus 40/20 and 45/25, and Table 6 gives those two tasks the same instructions as base T1 and T4 in Table 5. Section 5 states that the adaptation tasks share movement components with the base tasks. Table 2 therefore records a small, uneven difference on reused skills, concentrated on two training instructions in a new layout.
The novel piece is not what the success tables measure. Section 2.2 locates the difference from prior latent-skill work in two places: quantized skills already support few-shot transfer with a frozen decoder, and villa-X already factorizes latent planning from action generation. The manuscript’s own distinction is the shared movement code plus Latent Override, which is meant to stop an observation-conditioned world–action model from ignoring its latent (Section 2.2). Table 4a is the only measurement of that distinction, and it is an action L1 of 0.1087 versus 0.1535, not task success. Tables 1–3 instead compare a small prior head, with the 2B world–action model frozen, against full-model fine-tuning of Cosmos-Policy and π0.5 for 500 steps (Section 4.1, Appendix C). A small module trained for the same step budget will also be cheaper (Table 3: 8.2 GB on one GPU versus 80 GB and 93 GB on four) and can match success without the codebook being the cause. There is no control that keeps this frozen world–action model and replaces the atomic-movement prior with a similarly small language-conditioned adapter, and no comparison to the frozen-decoder or latent-planning baselines cited in Section 2.2. The reported adaptation result is therefore what parameter-efficient use of a strong backbone would be expected to show. It does not establish a new movement interface.
Questions for authors
Under the Table 2 budget, freeze this same world–action model and adapt a language-conditioned module with the same number of trainable parameters, predicting actions with no atomic-movement codebook. Report per-task success next to Table 2. If that module matches 58.33%, the adaptation result does not depend on the movement interface.
3 comments
Dear Reviewer r2nw, thank you again for your time and thoughtful feedback. We hope that our additional clarification and experiments have addressed the concerns and questions raised in your review. We would greatly appreciate it if you could let us know whether your concerns have been addressed. If you have further questions, we would be very happy to discuss.
Thanks the authors for the detailed reply. My concerns are largely addressed and I am happy to raise my score.
Response to Reviewer r2nw
We thank the reviewer for the constructive feedbacks and suggestions. We are glad that the reviewer finds our latent override design meaningful, and acknowledges our real-world experiments. We will then address the reviewer's remaining concerns.
W1.
Our evaluation tests primarily whether learned movements can support new objects, layouts, and task sequences with only 6–8 demonstrations and a frozen execution policy. Reusing movement components is central to this goal. Keeping instructions unchanged while changing layouts isolates spatial generalization: the same goal requires a different trajectory. On these novel arrangement tasks, our average success is 50%, versus 30% for Cosmos-Policy and 35% for π0.5. Across all six tasks, ours achieves the highest observed average while updating only the prior adapter. We do not claim a uniform advantage, yet the results show competitive adaptation with substantially lower training cost. Each task uses 20 trials.
W2/Q1.
We thank the reviewer for the valuable feedback. First, we further conduct an experiment measuring our latent override design's actual impact on real-robot experiments:
The results show that, without latent override, success rate is comparable to or below Cosmos-Policy baseline on all base tasks. In the deloyment we observe that the policy trained without latent override is 'reciting trajectories' seen during training stage purely based on visual observation, regardless of what the latent prior outputs. This verifies that the latent override design is vital to train a usable atomic movement-conditioned policy.
Second, following the reviewer's suggestion on comparing against a language-conditioned adapter, we add Cosmos-Policy + LoRA (using rank 8, same as our latent prior) under the same few-shot protocol (the same few-shot demonstrations and 500 adaptation steps ) and explicitly compare the compute resources used.
The result shows that with LoRA finetuning, Cosmos-Policy performs worse than baseline, because model has to learn by itself from the very limited few-shot demonstrations. Given the novelity of the new tasks, this learning process could be quite difficult. Our method, on the other hand, solves this problem through a highly composable vocabulary, boosting the learning of new tasks.
We also compare the efficiency of different methods:
The results show that the LoRA version still costs significantly more expensive than our method, primarily due to the fact that the full WAM model still needs to be loaded and forwarded, while our method only requires loading and updating a much smaller model.