- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
UniLAP learns a language-conditioned continuous latent-action policy from action-free human and robot videos. A frozen DreamDojo latent-action encoder provides posterior targets from frame transitions, while a flow-matching student predicts latent-action chunks using only the current observation and instruction. The paper introduces a flow-adaptive posterior-distillation objective and correspondence-based visual residual supervision. An embodiment-specific decoder, together with policy adaptation, subsequently maps these latents to executable actions. Evaluation covers human hand-pose forecasting, single-arm and humanoid manipulation, and real-world deployment.
Critical strengths
- The central idea is promising and practically relevant. Learning a generative policy over continuous latent actions offers a reasonable way to exploit action-free videos while retaining an actionable interface for downstream adaptation. Separating video-based pretraining from embodiment-specific action grounding is useful, and the proposed system demonstrates a workable implementation across substantially different action spaces.
- The paper provides meaningful empirical and diagnostic evidence. Evaluation extends beyond a single manipulation benchmark. On EgoDex, improvements hold for single-sample forecasting as well as best-of-multiple-sample evaluation. The downstream results and pretraining ablations support the usefulness of the learned initialization. Action probes, language perturbations, and sampling analyses also provide evidence that the latent representation contains behavior-relevant information.
Critical weaknesses
-
The contribution relative to existing latent-action policies and the inherited teacher representation is not sufficiently isolated.
Continuous latent actions and causal latent-action policies already have substantial precedent. CLAM learns an observation-conditioned continuous latent policy, while villa-X uses a generative latent-action actor conditioned on the current observation and language. Consequently, the contribution needs to be distinguished more precisely from these existing formulations. The statement that discrete representations cannot describe multimodal behavior is also too strong: quantization limits resolution and capacity, but a distribution over discrete codes can still be multimodal. CLAM, villa-X.
UniLAP appears to contribute a particular combination of continuous-teacher distillation, generative latent-policy learning, and auxiliary supervision. However, the experiments do not adequately separate these factors. In particular, replacing DreamDojo with a UniVLA VQ teacher changes the teacher’s training history, architecture, representation quality, and quantization simultaneously. This establishes sensitivity to teacher choice, but does not isolate continuity as the source of improvement. A comparison using the same teacher representation with and without quantization would be considerably more informative.
-
The generality of the framework and the claimed latent properties remain only partially validated.
The approach is instantiated primarily in a fixed DreamDojo latent space. It remains unclear how much of its effectiveness depends on this particular representation, and whether the proposed recipe transfers to another continuous latent-action teacher or to features from a different video/world model. Teacher dependence is not inherently a weakness, but the paper should distinguish a generally applicable learning framework from a successful DreamDojo-based instantiation. Better video-generation quality also does not automatically imply a better action representation.
The latent diagnostics are useful but support narrower conclusions than some of the claims. Linear probes demonstrate action-predictive information; language-distance measurements demonstrate sensitivity; and endpoint clustering provides evidence of distributional structure. They do not directly establish cross-embodiment semantic alignment, correct behavioral responses to instruction changes, or the validity of multiple sampled behaviors. Similarly, predicting correspondence-aligned feature residuals does not by itself establish invariance to appearance or camera motion. Since downstream adaptation usually updates the policy, the paper should also clarify which properties already exist after action-free pretraining and which emerge through action supervision.
Questions for authors
- Contribution clarification: What is the precise advance over existing continuous latent policies and generative latent-action actors? Which benefits are attributed specifically to continuous representation, posterior distillation, and correspondence supervision? A matched-teacher quantization comparison would directly address an important part of this question.
- Teacher dependence and generality: Which parts of the method require DreamDojo’s Gaussian posterior, and which apply to arbitrary continuous transition representations? Have you tested another continuous teacher or a reasonable alternative video-feature target under the same student and adaptation protocol? Such evidence would help determine whether the contribution extends beyond the selected teacher.
- Representation validation: Do the robotic latent probes use the Stage I checkpoint or the policy after action-supervised adaptation? Can you separate these stages and establish how much downstream control relies on the latent, given the decoder’s direct access to visual features and proprioception? Latent removal or shuffling would provide a particularly useful control.
I find the underlying idea interesting, and my current score reflects unresolved questions about contribution and evidence rather than a fundamental objection to the approach. I am open to increasing my score if the response clearly establishes the distinction from prior latent-action policies and provides convincing clarification or targeted evidence concerning teacher dependence and the claimed latent properties. I would place greater weight on these focused controls than on additional leaderboard comparisons.
4 comments
W1-W2
4. Scope of latent diagnostics and multimodality claims (W1, W2)
Our continuous formulation addresses the target-resolution bottleneck of finite codebooks. A distribution over discrete codes can also be multimodal.
Frozen Stage I action probes show higher held-out action than the visual reference (Appendix D.1, Table 10 of the original submission). Language perturbations show sensitivity to verb and location changes and relative stability under paraphrasing (Figure 7). These probes do not by themselves establish cross-embodiment semantic alignment. After adaptation, xArm7 succeeds on 23 of 50 rollouts for one unseen compositional instruction. The latent intervention in Q3 further shows that the adapted controller relies on latent content.
Our conditional policy can represent multimodal action distributions. The decoded-endpoint diagnostic rejects the matched Gaussian reference for 238/512 observations, compared with 2/100 and 1/100 Gaussian controls. This supports departures from that reference but does not exclude non-Gaussian unimodality. Best-of- ADE decreases from 0.034 m at to 0.029 m at , showing improved best-candidate trajectory accuracy with additional sampling.
The added motion supervision target comparison table in the revised manuscript shows higher action-prediction with the residual target than with future features or matched future features, including under high camera motion. The average increase in action-prediction ADE across the tested brightness and contrast perturbations was below 0.1%. Together with the original motion ablation (Figure 7), these results support action-relevant motion supervision and stability under the tested appearance changes.
Thank you for the additional controls. The comparison using the same frozen DreamDojo teacher addresses my main concern about the original quantization experiment. The 11.1 percentage point advantage provides direct evidence for retaining continuous targets under the tested setting.
The clarification that the action probes use the frozen Stage I policy helps distinguish what is learned during pretraining from what emerges during adaptation. The Stage II intervention also directly addresses my question about whether the decoder uses the latent when visual and proprioceptive inputs remain available. Generalization to a second continuous teacher remains open, but these results address my main concerns and I am raising my score from 2 to 3.
We sincerely appreciate the time and care you took to revisit our work. Thank you for following up.
Q1-Q3
1. Contribution relative to prior latent-action policies (W1, Q1)
Our goal is to learn transferable motion priors from large amounts of human video for robot control. UniLAP pretrains a shared continuous latent action policy with flow-adaptive posterior distillation and motion supervision. Robot action labels are introduced only during adaptation.
CLAM first uses its LAM to extract latent actions from task demonstrations. For downstream tasks, it trains a separate policy for each task on each embodiment. In contrast, UniLAP pretrains one shared latent action policy and adapts it to each robot embodiment by finetuning the LAP.
Villa-X uses VQ-derived latent targets, introducing the granularity bottleneck discussed in our Introduction. UniLAP models fine-grained latent variations without this quantization step.
For this rebuttal, we compared continuous and 16-bit product-quantized latents from the same frozen DreamDojo teacher under matched robot-only training.
Continuous latents achieve 11.1 percentage points higher success under this PQ setting.
With DreamDojo fixed, our full objective improves WidowX success by 4.1 percentage points over removing posterior regularization and 5.5 points over constant weighting (Table 4). Motion supervision from visual correspondences reduces early-training EgoDex FDE by 4.6% to 5.4% and improves control (Figure 7).
2. Teacher dependence and generality (W2, Q2)
UniLAP requires a LAM teacher to provide latent-action targets for video transitions. Flow matching uses target samples, while posterior distillation additionally uses a mean and covariance. Other continuous transition representations can be used through a compatible interface, with a fixed covariance for regularization when needed. Motion supervision and action decoding do not require teacher posterior statistics.
We chose DreamDojo as our continuous LAM teacher for its large-scale pretraining and the strong results reported in its paper. We therefore focused our evaluation on DreamDojo and have not yet tested another continuous teacher under the same protocol.
We have evaluated UniVLA’s discrete LAM as an alternative teacher (Table 4 of the original submission). Its codebook vectors provide the latent targets, and we use a fixed covariance for distillation.
3. Stage I versus Stage II properties and latent reliance (W2, Q3)
The action probe (Appendix D.1 and Table 10 of the original submission) fits a linear regression model to predict recorded actions from latent actions and reports R2 on held-out data. It measures the linear predictability of actions from the learned representation.
The probe uses only the frozen Stage I policy, before action-supervised Stage II adaptation. Action labels train the linear regression model, while LAP parameters remain fixed. The resulting R2 exceeds the DINOv2 reference on EgoDex, Bridge and Fractal.
For this rebuttal, we conducted a new latent-input intervention on an adapted Stage II model. We kept the model fixed and retained its visual and proprioceptive inputs, changing only the latent actions passed to the decoder. We replaced the whole latent-action chunk with one generated by the same policy for another trajectory, or set it to zero. All conditions used the same three WidowX tasks and matched initial conditions.
These results show that the adapted Stage II controller relies on the latent content.