acceptodds
Under review as a conference paper at ICLR 2027

UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation

Abstract

Latent Action Models (LAMs) can capture manipulation priors from action-free videos, but discrete latents usually impose a granularity bottleneck, limiting their ability to represent fine-grained action. Recent continuous alternatives mitigate this, but they are typically used as proxies for world modeling and require a further planning step to produce executable actions. We thus introduce UniLAP, a unified continuous latent-action policy that models the latent space via flow matching and bypasses reliance on future frames during inference. By distilling a LAM teacher with a flow-adaptive objective, UniLAP inherits the teacher's action representation while retaining the multimodal distribution of feasible behaviors. To decouple latent representations from task-irrelevant appearance, we extract a label-free motion target to isolate relevant kinematic cues from spurious visual dynamics. With the established embodiment-agnostic latent manifold, UniLAP can be easily adapted to novel platforms. Pretrained on multi-source videos at a compact 1.1B parameter scale, UniLAP outperforms state-of-the-art latent-action and Vision-Language-Action (VLA) baselines across hand pose forecasting, single-arm and humanoid manipulation, and real-world deployment.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Latent Action, Action-Free Video, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Dra1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniLAP learns a language-conditioned continuous latent-action policy from action-free human and robot videos. A frozen DreamDojo latent-action encoder provides posterior targets from frame transitions, while a flow-matching student predicts latent-action chunks using only the current observation and instruction. The paper introduces a flow-adaptive posterior-distillation objective and correspondence-based visual residual supervision. An embodiment-specific decoder, together with policy adaptation, subsequently maps these latents to executable actions. Evaluation covers human hand-pose forecasting, single-arm and humanoid manipulation, and real-world deployment.

Critical strengths
  1. The central idea is promising and practically relevant. Learning a generative policy over continuous latent actions offers a reasonable way to exploit action-free videos while retaining an actionable interface for downstream adaptation. Separating video-based pretraining from embodiment-specific action grounding is useful, and the proposed system demonstrates a workable implementation across substantially different action spaces.
  2. The paper provides meaningful empirical and diagnostic evidence. Evaluation extends beyond a single manipulation benchmark. On EgoDex, improvements hold for single-sample forecasting as well as best-of-multiple-sample evaluation. The downstream results and pretraining ablations support the usefulness of the learned initialization. Action probes, language perturbations, and sampling analyses also provide evidence that the latent representation contains behavior-relevant information.
Critical weaknesses
  1. The contribution relative to existing latent-action policies and the inherited teacher representation is not sufficiently isolated.

    Continuous latent actions and causal latent-action policies already have substantial precedent. CLAM learns an observation-conditioned continuous latent policy, while villa-X uses a generative latent-action actor conditioned on the current observation and language. Consequently, the contribution needs to be distinguished more precisely from these existing formulations. The statement that discrete representations cannot describe multimodal behavior is also too strong: quantization limits resolution and capacity, but a distribution over discrete codes can still be multimodal. CLAM, villa-X.

    UniLAP appears to contribute a particular combination of continuous-teacher distillation, generative latent-policy learning, and auxiliary supervision. However, the experiments do not adequately separate these factors. In particular, replacing DreamDojo with a UniVLA VQ teacher changes the teacher’s training history, architecture, representation quality, and quantization simultaneously. This establishes sensitivity to teacher choice, but does not isolate continuity as the source of improvement. A comparison using the same teacher representation with and without quantization would be considerably more informative.

  2. The generality of the framework and the claimed latent properties remain only partially validated.

    The approach is instantiated primarily in a fixed DreamDojo latent space. It remains unclear how much of its effectiveness depends on this particular representation, and whether the proposed recipe transfers to another continuous latent-action teacher or to features from a different video/world model. Teacher dependence is not inherently a weakness, but the paper should distinguish a generally applicable learning framework from a successful DreamDojo-based instantiation. Better video-generation quality also does not automatically imply a better action representation.

    The latent diagnostics are useful but support narrower conclusions than some of the claims. Linear probes demonstrate action-predictive information; language-distance measurements demonstrate sensitivity; and endpoint clustering provides evidence of distributional structure. They do not directly establish cross-embodiment semantic alignment, correct behavioral responses to instruction changes, or the validity of multiple sampled behaviors. Similarly, predicting correspondence-aligned feature residuals does not by itself establish invariance to appearance or camera motion. Since downstream adaptation usually updates the policy, the paper should also clarify which properties already exist after action-free pretraining and which emerge through action supervision.

Questions for authors
  1. Contribution clarification: What is the precise advance over existing continuous latent policies and generative latent-action actors? Which benefits are attributed specifically to continuous representation, posterior distillation, and correspondence supervision? A matched-teacher quantization comparison would directly address an important part of this question.
  2. Teacher dependence and generality: Which parts of the method require DreamDojo’s Gaussian posterior, and which apply to arbitrary continuous transition representations? Have you tested another continuous teacher or a reasonable alternative video-feature target under the same student and adaptation protocol? Such evidence would help determine whether the contribution extends beyond the selected teacher.
  3. Representation validation: Do the robotic latent probes use the Stage I checkpoint or the policy after action-supervised adaptation? Can you separate these stages and establish how much downstream control relies on the latent, given the decoder’s direct access to visual features and proprioception? Latent removal or shuffling would provide a particularly useful control.

I find the underlying idea interesting, and my current score reflects unresolved questions about contribution and evidence rather than a fundamental objection to the approach. I am open to increasing my score if the response clearly establishes the distinction from prior latent-action policies and provides convincing clarification or targeted evidence concerning teacher dependence and the claimed latent properties. I would place greater weight on these focused controls than on additional leaderboard comparisons.

4 comments

1. Contribution relative to prior latent-action policies (W1, Q1)

Our goal is to learn transferable motion priors from large amounts of human video for robot control. UniLAP pretrains a shared continuous latent action policy with flow-adaptive posterior distillation and motion supervision. Robot action labels are introduced only during adaptation.

CLAM first uses its LAM to extract latent actions from task demonstrations. For downstream tasks, it trains a separate policy for each task on each embodiment. In contrast, UniLAP pretrains one shared latent action policy and adapts it to each robot embodiment by finetuning the LAP.

Villa-X uses VQ-derived latent targets, introducing the granularity bottleneck discussed in our Introduction. UniLAP models fine-grained latent variations without this quantization step.

For this rebuttal, we compared continuous and 16-bit product-quantized latents from the same frozen DreamDojo teacher under matched robot-only training.

DreamDojo latent representationWidowX success rate
Continuous41.7%
Product-quantized (16-bit)30.6%

Continuous latents achieve 11.1 percentage points higher success under this PQ setting.

With DreamDojo fixed, our full objective improves WidowX success by 4.1 percentage points over removing posterior regularization and 5.5 points over constant weighting (Table 4). Motion supervision from visual correspondences reduces early-training EgoDex FDE by 4.6% to 5.4% and improves control (Figure 7).

2. Teacher dependence and generality (W2, Q2)

UniLAP requires a LAM teacher to provide latent-action targets for video transitions. Flow matching uses target samples, while posterior distillation additionally uses a mean and covariance. Other continuous transition representations can be used through a compatible interface, with a fixed covariance for regularization when needed. Motion supervision and action decoding do not require teacher posterior statistics.

We chose DreamDojo as our continuous LAM teacher for its large-scale pretraining and the strong results reported in its paper. We therefore focused our evaluation on DreamDojo and have not yet tested another continuous teacher under the same protocol.

We have evaluated UniVLA’s discrete LAM as an alternative teacher (Table 4 of the original submission). Its codebook vectors provide the latent targets, and we use a fixed covariance for distillation.

3. Stage I versus Stage II properties and latent reliance (W2, Q3)

The action probe (Appendix D.1 and Table 10 of the original submission) fits a linear regression model to predict recorded actions from latent actions and reports R2R^2 on held-out data. It measures the linear predictability of actions from the learned representation.

The probe uses only the frozen Stage I policy, before action-supervised Stage II adaptation. Action labels train the linear regression model, while LAP parameters remain fixed. The resulting R2R^2 exceeds the DINOv2 reference on EgoDex, Bridge and Fractal.

For this rebuttal, we conducted a new latent-input intervention on an adapted Stage II model. We kept the model fixed and retained its visual and proprioceptive inputs, changing only the latent actions passed to the decoder. We replaced the whole latent-action chunk with one generated by the same policy for another trajectory, or set it to zero. All conditions used the same three WidowX tasks and matched initial conditions.

Decoder latent inputSuccessful rollouts
Normal27/72
Chunk from another trajectory0/72
Zero0/72

These results show that the adapted Stage II controller relies on the latent content.

4. Scope of latent diagnostics and multimodality claims (W1, W2)

Our continuous formulation addresses the target-resolution bottleneck of finite codebooks. A distribution over discrete codes can also be multimodal.

Frozen Stage I action probes show higher held-out action R2R^2 than the visual reference (Appendix D.1, Table 10 of the original submission). Language perturbations show sensitivity to verb and location changes and relative stability under paraphrasing (Figure 7). These probes do not by themselves establish cross-embodiment semantic alignment. After adaptation, xArm7 succeeds on 23 of 50 rollouts for one unseen compositional instruction. The latent intervention in Q3 further shows that the adapted controller relies on latent content.

Our conditional policy can represent multimodal action distributions. The decoded-endpoint diagnostic rejects the matched Gaussian reference for 238/512 observations, compared with 2/100 and 1/100 Gaussian controls. This supports departures from that reference but does not exclude non-Gaussian unimodality. Best-of-SS ADE decreases from 0.034 m at S=1S=1 to 0.029 m at S=10S=10, showing improved best-candidate trajectory accuracy with additional sampling.

The added motion supervision target comparison table in the revised manuscript shows higher action-prediction R2R^2 with the residual target than with future features or matched future features, including under high camera motion. The average increase in action-prediction ADE across the tested brightness and contrast perturbations was below 0.1%. Together with the original motion ablation (Figure 7), these results support action-relevant motion supervision and stability under the tested appearance changes.

Thank you for the additional controls. The comparison using the same frozen DreamDojo teacher addresses my main concern about the original quantization experiment. The 11.1 percentage point advantage provides direct evidence for retaining continuous targets under the tested setting.

The clarification that the action probes use the frozen Stage I policy helps distinguish what is learned during pretraining from what emerges during adaptation. The Stage II intervention also directly addresses my question about whether the decoder uses the latent when visual and proprioceptive inputs remain available. Generalization to a second continuous teacher remains open, but these results address my main concerns and I am raising my score from 2 to 3.

We sincerely appreciate the time and care you took to revisit our work. Thank you for following up.

Reviewer_XDQ8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes UniLAP, a continuous latent-action policy pretrained on action-free human and robot videos. A conditional flow-matching student predicts latent action chunks from the current observation and language instruction, using a frozen DreamDojo latent-action encoder as a teacher. Pretraining combines flow matching, a timestep-weighted posterior-mean regularizer, and an auxiliary target derived from cross-frame DINOv2 correspondences. An embodiment-specific decoder is subsequently trained on labeled actions, with policy finetuning for robot embodiments.

Evaluation covers EgoDex hand-pose forecasting, SimplerEnv, RoboCasa-GR1, and a physical xArm7. The results show improved hand forecasting and competitive manipulation performance with a reported 1.1B-parameter model, although some gains are small and the evidence for the proposed mechanisms remains incomplete.

Critical strengths
  1. The paper provides a useful way to transfer latent-action representations inferred with future frames during training into a policy that operates without future frames at deployment. The student samples an entire latent chunk conditioned only on the current observation and instruction, and the downstream decoder handles different action spaces. This is a practically relevant integration of video pretraining and robot policy learning.
  2. The evaluation spans substantially different embodiments and includes a physical platform absent from Stage I. On EgoDex, UniLAP improves single-sample ADE from the strongest listed baseline’s 0.039 m to 0.034 m, and achieves a best-of-10 ADE of 0.029 m (Table 1). The pretraining and data-mixture ablations also provide evidence that video pretraining and combining human and robot data are useful (Table 4 and Figure 7). These results make the approach worth investigating further.
Critical weaknesses

1.The interpretation of the correspondence residual as motion supervision needs stronger support. Equations (5)–(6) align future features to current patches and subtract their features, without explicitly encoding spatial displacement. With accurate matching and stable descriptors, the residual may be small despite object motion, while appearance changes or matching errors may produce nonzero residuals. Figure 7 demonstrates an auxiliary-loss benefit but does not establish motion disentanglement or camera-motion robustness. Comparisons against simpler feature differences or correspondence-based displacement would strengthen this central claim.

2.The experiments do not fully isolate or establish the claimed advantages. The DreamDojo–UniVLA comparison changes the teacher as well as the representation, so it cannot attribute the gain specifically to continuous latents. Several performance margins are also modest: 42.7% versus 42.3% on RoboCasa-GR1 and 28.0% versus 22.7% on the physical robot. The three-task WidowX ablation gain of 44.4% versus 40.3% amounts to approximately three additional successes in 72 trials, without training-seed variability or uncertainty estimates. Moreover, most simulated target embodiments appear in Stage I, limiting the evidence for transfer to unseen embodiments.

Questions for authors

1.How does the aligned feature residual distinguish spatial motion from appearance changes or matching errors? Please clarify how step-h targets indexed at (t+h) align with predictions based on (f_t), and provide a controlled comparison with simpler motion targets.

2.Can continuous and quantized latents be compared using a common teacher and matched training settings? How is the discrete UniVLA teacher mapped to the Gaussian mean and covariance required by Equation (4)?

3.Can you report training-seed variability and uncertainty estimates for the main manipulation results, and clarify which baselines were evaluated under identical settings? Please also confirm that Stage I and the frozen teacher exclude EgoDex test trajectories.

4.How does the Gaussian-null test in Appendix D.2 distinguish multimodality from non-Gaussian unimodality? Does the posterior-mean regularizer in Equation (4) affect the flow-matching target distribution or mode coverage?

4 comments

1. Feature residuals, temporal alignment, and simpler motion targets (W1, Q1)

The motion predictor ϕmot\phi_{\text{mot}} is designed to encourage latent actions to encode motion-relevant information. It takes current observation features as visual input and uses student latent actions as conditioning to predict correspondence-aligned feature changes. The prediction loss backpropagates into the student through this conditioning path. The motion ablation in the original submission shows 4.6% to 5.4% lower early-training EgoDex FDE and improved WidowX control.

The residual provides motion supervision in feature space without explicitly separating spatial motion, appearance changes and matching errors.

For this rebuttal, we conducted new experiments comparing frozen Stage I students trained with three prediction targets. For each representation, we trained a linear regression model to predict wrist and fingertip displacements using 512 EgoDex training trajectories and evaluated it on 256 separate test trajectories. Camera-motion groups were defined using thresholds set on the training split.

Representation or training targetAll cases R2R^2 ↑High camera motion R2R^2 ↑
Correspondence residual (ours)0.042080.04589
Future features0.039880.04303
Matched future features, without subtraction0.039990.04288
DINOv2, visual-only reference−0.00397−0.00271

The latent models use the current image and language, while DINOv2 uses the image alone. Our latents achieve higher action-prediction R2R^2 than both alternative-target students in both groups. This supports the value of our target for retaining action-relevant information, including in cases with high camera motion.

We also tested appearance sensitivity by changing brightness and contrast with factors of 0.8 and 1.2. With the linear regression model trained on clean data and held fixed, our mean ADE increased only from 54.580 to 54.621 mm, an increase of 0.041 mm (0.074%). Action prediction from our latents therefore remains stable under these tested appearance changes.

To examine whether the motion predictor uses the latent condition, we held the current image and prediction target fixed and replaced the latent condition with one from another trajectory, following the predictor’s training path. Prediction loss increased by 0.0186, 0.0281 and 0.0286 at flow times 0.1, 0.5 and 0.9, respectively. This shows that the predictor uses the student latent condition to predict the feature changes.

Our predictor performs conditional prediction of feature changes over future intervals given current observation features and latent actions. The future time index specifies which interval to predict. Future frames provide the training targets, and the prediction loss encourages latent actions to capture information needed to explain these changes.

2. Fixed-teacher quantization and the UniVLA Gaussian interface (W2, Q2)

For this rebuttal, we added a comparison between continuous and quantized latents from the same frozen DreamDojo teacher. Both branches use robot-only data and the same training settings. Product quantization uses four codebooks with 16 entries each, giving a nominal 16-bit budget.

DreamDojo representation, both branches robot-onlyWidowX success rate
Continuous latent41.7%
Post-hoc product-quantized latent30.6%

Both use the three-task WidowX protocol with 24 rollouts per task. The 11.1-percentage-point advantage supports the continuous interface under this PQ configuration. The original DreamDojo/UniVLA comparison evaluates teacher choice.

The quantized DreamDojo and UniVLA branches both use codebook vectors as flow-matching targets and distillation anchors. In the DreamDojo PQ branch, sampled latents are quantized before action decoding. The DreamDojo PQ branch retains the continuous teacher’s variance-based regularization weights.

For UniVLA, we concatenate four 128-dimensional codebook vectors into a 512-dimensional target and anchor zVQ,hz_{\mathrm{VQ},h}. We use a fixed isotropic covariance Σh=σ02I\Sigma_h=\sigma_0^2 I, with log⁡σ02=−3.6922\log\sigma_0^2=-3.6922, calibrated to match DreamDojo’s mean inverse variance under the training-data sampling distribution. Equation (4) becomes

=\mathbb E\left[\frac{\tau}{2\sigma_0^2} \left\|\hat z_{\tau,h}-z_{\mathrm{VQ},h}\right\|_2^2\right].$$ These covariance terms weight the distillation penalty. UniVLA itself provides no sample-specific Gaussian uncertainty. The revised appendix specifies the UniVLA interface.

3. Training variability, uncertainty, baseline protocols, and test exclusion (W2, Q3)

Our evaluation follows StarVLA's SimplerEnv protocol and the RoboCasa-GR1 protocol used by GR00T and StarVLA, using the same benchmark environments and corresponding tasks. SimplerEnv's Visual Matching and Variant Aggregation results are reported separately.

For the EgoDex baselines, we follow the official implementations accompanying the X-IL paper. Simulation baseline values are taken from the cited papers. UniLAP and π0.5\pi_{0.5} are evaluated on the same xArm7 platform under a common protocol.

As in prior work, we report manipulation success rates without training-seed variability estimates. On EgoDex, we additionally report best-of-SS results for S=1,5,10S=1,5,10, with ADEs of 0.034, 0.031 and 0.029 m, respectively.

Stage I training and official EgoDex test trajectory identifiers are disjoint. The frozen teacher's training metadata lists EgoDex parts 1 to 5, with no official test paths.

4. Multimodality testing and the effect of posterior regularization (Q4)

The Gaussian-null diagnostic compares the 2-means separation of decoded endpoints with a covariance-matched Gaussian reference. It cannot by itself distinguish multimodality from every non-Gaussian unimodal distribution, and non-rejected observations remain inconclusive.

For 512 EgoDex observations, we draw 64 latents each and decode them with the same frozen deterministic decoder. 238 observations (46.5%) reject the Gaussian null, compared with 2/100 and 1/100 Gaussian controls processed through the same PCA and clustering pipeline. Among rejected observations, median centroid separation is 4.53 cm.

UniLAP models potentially multimodal action distributions conditioned on the current observation and instruction. Marginalizing over unobserved futures yields a mixture of teacher posteriors that permits multiple modes. On EgoDex, best-of-SS ADE improves from 0.034 m at S=1S=1 to 0.029 m at S=10S=10, showing that additional conditional samples provide more accurate trajectory candidates.

Posterior regularization leaves the teacher targets and their sampling distribution unchanged, but changes the training objective and can affect the learned distribution and mode coverage. Each observed video transition has its own mean anchor μh\mu_h. The flow-adaptive weight τ\tau reduces mean-alignment pressure early in the flow relative to constant weighting.

The original submission evaluates this design with DreamDojo held fixed.

Posterior regularizationWidowX success rate
Without posterior regularization, retaining motion loss40.3%
Constant weighting38.9%
Flow-adaptive weighting (full)44.4%

With the same teacher, flow-adaptive regularization improves success by 4.1 percentage points over removing the regularizer and 5.5 points over constant weighting.

5. Transfer to unseen embodiments (W2)

Our cross-embodiment transfer results include adaptation to an unseen robot and transfer between human and robot experience. The xArm7 platform is not included in the student's Stage I pretraining data. We evaluated UniLAP on this unseen platform through real-world experiments after supervised adaptation.

The data-mixture ablations in Table 4 of the original submission provide complementary evidence. A student pretrained only on EgoDex human videos adapts to WidowX and achieves 34.7% success. A robot-only pretrained student also supports EgoDex hand forecasting after action-decoder adaptation. Here, data-source labels refer to student Stage I pretraining.

Mixed pretraining improves both downstream evaluations over either source alone. WidowX, Google Robot and GR1 appear in mixed Stage I training. The ablation transferring from human videos to robot control and the xArm7 adaptation complement these results, supporting a shared LAP that adapts video priors to different action spaces.

Thank you for running the additional experiments. My main concern with the original comparison was that it changed both the teacher and the latent representation. Using the same frozen DreamDojo teacher addresses that concern directly. On the three WidowX tasks, the continuous latents achieved a 41.7% success rate, compared with 30.6% for the 16-bit product-quantized latents. This gives me a clearer reason to favor the continuous representation in this setting.

I still have some reservations about the interpretation of the motion results. The residual target gives a small improvement in predicting hand displacement, including in clips with substantial camera motion. How much camera motion the representation retains remains an open question. I would also interpret the smaller differences in manipulation success rates cautiously, since variability across training runs was not reported.

Overall, the fixed-teacher comparison resolves the concern that weighed most heavily in my original review. I am raising my rate.

Thank you very much for your thoughtful follow-up and for taking the time to review the additional experiments.

Reviewer_DBi5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UniLAP, which trains a latent action policy in a continuous latent action space. During training, an auxiliary LdistillL_{distill} loss maintains alignment between the student and teacher representations, while LmotionL_{motion} encourages the student’s predicted latent actions to capture useful motion information. The trained latent action policy can be paired with different action decoders to enable cross-embodiment action generation.

Critical strengths

The evaluation of latent action representations is relatively thorough. The latent action policy is tested on several downstream tasks after post-training, including hand pose forecasting and simulated manipulation in SimplerEnv, as well as future-frame reconstruction. These results indicate that the latent actions capture information about dynamics.

Critical weaknesses
  1. Limited novelty. Using continuous latent actions as supervision targets for latent action policies has already been explored in CLAM and LatentVLA. The main methodological addition appears to be two auxiliary loss terms for continuous latent action learning, yet the ablation results suggest that their benefits are marginal.
  2. Insufficient evidence of cross-embodiment capabilities. The paper uses the pretrained LAP as an initialization for separate post-training on different robots. This does not establish the LAP’s cross-embodiment capabilities, such as joint training on data from multiple embodiments and deployment across them.
  3. The real-world tasks are too simple to adequately demonstrate the model’s capabilities.
Questions for authors

Could the authors clarify how this work fundamentally differs from prior continuous latent action approaches, such as CLAM and LatentVLA? Based on the current results, it remains unclear whether the performance gains primarily stem from the pretrained DreamDojo LAM or from the proposed method.

[1] Anthony Liang, Pavel Czempin, Matthew M. Hong, Yutai Zhou, Jingzhen Wang, Erdem Bıyık, and Stephen Tu. “CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations.” In 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026. arXiv:2505.04999.

[2] Junming Wang. “LatentVLA: Taming Latent Space for Generalizable and Long-Horizon Bimanual Manipulation.” Proceedings of the AAAI Conference on Artificial Intelligence, 40(22): 18593–18601, 2026. DOI: 10.1609/aaai.v40i22.38926.

3 comments

1. Novelty and method differences (W1, author question)

Our goal is to learn transferable motion priors from large amounts of human video for robot control.

UniLAP pretrains a shared latent action policy directly on action-free human and robot videos. Flow-adaptive posterior distillation constrains predicted latents using the teacher’s posterior mean and covariance with flow-time weighting. Correspondence-based motion supervision encourages these latents to retain motion-relevant information. Robot action labels are introduced only during adaptation.

CLAM first uses its LAM to extract latent actions from task demonstrations. For downstream tasks, it trains a separate policy for each task on each embodiment. In contrast, UniLAP pretrains one shared latent action policy and adapts it to each robot embodiment by finetuning the LAP.

LatentVLA uses reconstruction and robot action supervision to train TA-LAM, then pretrains LADT on latent sequences from labeled robot data. UniLAP, however, directly uses action-free human videos to pretrain the policy, and learns a new decoder and finetunes the LAP during robot embodiment adaptation.

2. Method gains and attribution (W1, author question)

We chose DreamDojo as our continuous LAM teacher for its large-scale pretraining and the strong results reported in its paper. The robot-only teacher comparison in Table 4 gives 41.7% WidowX success with DreamDojo and 29.2% with the UniVLA VQ teacher.

For this rebuttal, we added a comparison between continuous and quantized latents from the same frozen DreamDojo teacher.

DreamDojo latent representation, robot-onlyWidowX success rate
Continuous41.7%
Product-quantized (16-bit)30.6%

With the teacher fixed, continuous latents achieve 11.1 percentage points higher success under this PQ setting. Together, these comparisons support our choice of DreamDojo's continuous latent interface.

We evaluate our learning objectives with DreamDojo held fixed. In Table 4, the full objective reaches 44.4% WidowX success, versus 40.3% without posterior regularization and 38.9% with constant weighting, corresponding to gains of 4.1 and 5.5 percentage points. The motion ablation in Figure 7 also reports 4.6% to 5.4% lower early-training EgoDex FDE and improved WidowX control. These fixed-teacher ablations support additional gains from our training design.

3. Cross-embodiment capabilities and the adaptation protocol (W2)

We agree with this distinction. UniLAP currently uses a shared pretrained LAP initialization followed by separate policy finetuning and action-decoder training for each robot embodiment. Our experiments do not demonstrate joint post-training and deployment of one controller across embodiments.

Our goal is to learn a shared motion prior from action-free human and robot videos, then use target-embodiment action labels for adaptation. Under the same Stage II recipe, shared video pretraining raises WidowX success from 33.3% to 44.4% (Figure 7). Adding human videos improves the robot-only result from 41.7% to 44.4% (Table 4). We also adapt to xArm7, a platform absent from the student's Stage I data, using 50 demonstrations per task. These results support the practical value of shared video pretraining for robot adaptation.

We have clarified this scope in the limitations of the revised manuscript. Extending UniLAP’s embodiment-specific decoder to condition on embodiment information and support heterogeneous action spaces is a key step toward joint post-training and deployment across embodiments.

4. Real-world task complexity (W3)

Our current real-world tasks demand precise alignment and carefully controlled release. Nesting thin-walled cups requires alignment within a tight placement tolerance. Cube stacking adds a further challenge, as a mistimed release can deform the soft upper cube on contact and cause it to rebound off the stack.

The original submission also evaluates multi-stage manipulation in RoboCasa-GR1 simulation, using a 29-dimensional humanoid action space. For example, the robot must pick up a bottle, place it inside a cabinet and close the cabinet. Success requires completing the full sequence. UniLAP achieves 42.7% average success across six tasks, with 50 rollouts per task.

In future work, we plan to extend our real-world evaluation to long-horizon tasks.

Reviewer_DBi5OpenReview ↗

Official Comment by reviewer DBi5

I thank the authors for their response and for clarifying how UniLAP differs from CLAM and LatentVLA. CLAM already uses a pretrained latent action model to assign continuous latent action labels to task demonstrations and trains policies on those labels. I recognize that UniLAP extends policy learning to shared pretraining across tasks. In my view, however, this distinction does not amount to a substantial methodological contribution over CLAM.

The ablations show that the proposed loss terms provide some benefit. However, the model does not demonstrate cross-embodiment capabilities, and I still consider the real-world tasks too simple. I therefore maintain my original rating.

Thank you for the follow-up and for clarifying your remaining concerns.

1. How UniLAP differs from CLAM

UniLAP is motivated by the scarcity of action-labeled robot demonstrations and the abundance of egocentric human videos and robot videos without action labels. We aim to learn transferable motion priors for robot control from these videos at scale. We pretrain a shared latent action policy (LAP) on these videos, then readily adapt it to a new robot with a small set of action-labeled demonstrations.

Beyond sharing a policy across tasks, UniLAP differs from CLAM in how it obtains the LAM teacher, pretrains the LAP, and adapts the LAP to new embodiments.

CLAMUniLAP
LAM teacherTrains its own LAM in a single-embodiment setting. Labeled robot actions jointly supervise the LAM and the action decoder.Uses the same publicly available, pretrained DreamDojo LAM as a frozen teacher for videos from humans and different robots.
LAP pretraining and adaptationUses its LAM to label expert demonstrations and trains a separate LAP for each target task. This training procedure provides no shared pretrained LAP for initializing policies across embodiments.Pretrains one shared LAP directly on human and robot videos without action labels, using flow-adaptive posterior distillation and correspondence-based motion supervision. A new robot starts from the same LAP and can be readily adapted with limited labeled demonstrations and a new action decoder.
How are tasks handled on one robot?A separate LAP is trained for each task.One language-conditioned LAP handles multiple tasks on the same robot.

Before embodiment-specific adaptation, linear readouts from the same frozen Stage I LAP predict held-out actions better than the visual-feature baseline on EgoDex, Bridge, and Fractal (Appendix D.1). Adding human videos to robot-video pretraining also raises WidowX success from 41.7% to 44.4% under the same adaptation protocol (Table 4).

2. Cross-embodiment capabilities

We agree that joint deployment across embodiments is an important direction, as discussed in the revised manuscript’s Limitations. Our experiments show that the same pretrained LAP can readily adapt to different robots with embodiment-specific decoders, including a physical xArm7 absent from Stage I. To address your concern, we conducted a preliminary experiment to explore closed-loop deployment of a single controller across two embodiments. We jointly fine-tuned one LAP and one shared decoder for WidowX and Google Robot. The decoder receives normalized proprioception and an embodiment ID and uses one output layer for both robots. We weight their training losses equally. For different state or action dimensions, we would pad states and action targets to common sizes, mask padded state inputs and action targets, and map valid outputs back to each robot’s native action space.

The supplement includes the decoder implementation and per-robot loss curves through 10k training steps. For both robots, joint and separate training show similar downward trends. Under the same setting, the joint 10k checkpoint achieved 58.03% on Google Robot Visual Matching versus 45.57% for the single-embodiment 10k checkpoint. On WidowX, the same joint checkpoint achieved 22.22% versus 30.56% for the single-embodiment 10k reference. The same jointly trained controller achieved successful closed-loop rollouts on both embodiments included in joint Stage II training.

3. Real-world task complexity

Our xArm7 experiments test whether a shared LAP pretrained on action-free videos can be adapted for physical manipulation on a robot absent from Stage I. The three tasks span placement with a forgiving release tolerance in Pineapple → Bowl and precision stacking in Stack Cups and Stack Cubes. We evaluated UniLAP and π0.5\pi_{0.5} on the same xArm7, on the same tasks, and under the same evaluation protocol. As reported in the Real-World Deployment paragraph of the original submission, UniLAP achieved 28.0% average success versus 22.7% for π0.5\pi_{0.5}, with successful rollouts on all three tasks. This evaluation shows that the video-pretrained LAP can be adapted for physical manipulation.

We agree that future real-world evaluations should include more complex, long-horizon tasks in which success depends on completing several interdependent stages.

Reviewer_iQ1iOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniLAP learns a continuous latent-action policy from action-free human and robot videos. A frozen DreamDojo teacher extracts latent-action posteriors from consecutive frames, while a flow-matching student learns to generate latent-action chunks from the current observation and language instruction alone. The method introduces a flow-adaptive posterior distillation loss and correspondence-based feature-residual supervision derived from frozen DINOv2 features. For downstream control, an embodiment-specific action decoder is trained, with the latent policy frozen or finetuned depending on the setting. Evaluation covers EgoDex hand-pose forecasting, SimplerEnv manipulation, RoboCasa-GR1 humanoid manipulation, and real-world xArm7 tasks. The results demonstrate the practical potential of transferring video-derived latent actions across different action spaces. However, the evidence does not fully establish the proposed motion supervision’s ability to separate camera and object motion, or isolate continuous representation as the source of improvement over a discrete teacher.

Critical strengths
  1. A complete route from action-free video learning to executable control. The method provides a practical pipeline for transferring continuous latent actions learned from action-free videos to executable robot control.
  2. Broad evaluation with supporting pretraining ablations. Experiments span hand-pose forecasting, single-arm and humanoid manipulation, and real-world deployment, with ablations supporting the benefit of pretraining.
Critical weaknesses
  1. Potential ambiguity of motion supervision on egocentric data. The proposed correspondence-based residual is used to introduce motion supervision. However, in egocentric data, visual changes can arise from both object motion and camera motion. It is unclear whether this objective distinguishes the two or inadvertently encourages the latent representation to encode camera movement alongside manipulation-relevant motion. This raises a concern about the suitability of the supervision for the EgoDex portion of pretraining, which the current experiments do not directly address.
  2. The continuous-versus-discrete comparison is confounded by teacher differences. The teacher ablation changes both the representation type and the teacher model, replacing DreamDojo with UniVLA. Differences in teacher pretraining, architecture, and capability may contribute to the performance gap, so the results do not isolate the benefit of continuous representation or establish quantization as the main bottleneck.
Questions for authors
  1. In egocentric videos, visual changes can arise from both camera motion and object motion. Could the proposed correspondence-derived residual encourage the latent actions to encode camera movement alongside manipulation-relevant motion? How does this affect pretraining on EgoDex, and do you have analyses or controlled experiments examining this potential ambiguity?
  2. The teacher ablation changes both the representation type and the teacher model, from DreamDojo to UniVLA. How do you separate the contribution of continuous representation from differences in teacher pretraining and model capability? Do you have a comparison that holds the teacher and training setup fixed while varying only whether its latent actions are quantized?
1 comment

1. Motion supervision and camera-motion ambiguity (W1, Q1)

The motion predictor ϕmot\phi_{\text{mot}} is designed to encourage latent actions to encode motion-relevant information. It predicts correspondence-aligned feature changes Δfh\Delta f_h from the current observation oto_t and latent action Z^τ\hat{Z}_\tau.

The motion ablation in the original submission shows 4.6% to 5.4% lower early-training EgoDex FDE and improved WidowX control.

Our objective does not explicitly separate camera and object motion. We focus on learning information useful for action prediction and control. Although UniVLA explicitly separates task-irrelevant and task-centric latents in its LAM, UniLAP has a higher success rate in our main WidowX comparison.

For this rebuttal, we conducted an action probe on frozen Stage I students trained with three motion prediction targets. For each representation, we trained a linear regression model to predict wrist and fingertip displacements using 512 EgoDex training trajectories and evaluated on 256 separate test trajectories. The high camera motion group contains 502 cases, identified from camera poses using thresholds set on the training split.

Representation or training targetAll cases R2R^2 ↑High camera motion R2R^2 ↑
Correspondence residual (ours)0.042080.04589
Future features0.039880.04303
Matched future features, without subtraction0.039990.04288
DINOv2, visual-only reference−0.00397−0.00271

All latent models use the current image and language instruction. DINOv2 uses the image alone. Our latents have the highest action R2R^2 among the three targets in both groups, including cases with high camera motion.

2. Continuous representation and the fixed-teacher quantization comparison (W2, Q2)

We added a comparison between continuous and quantized latents from the same frozen DreamDojo teacher.

Both models use the same robot-only data, the same flow-matching student and distillation loss, and 300k Stage I steps with a global batch size of 128. We quantize the 32-dimensional latent using product quantization (PQ) with four subvectors and 16 centers per subvector, for a 16-bit code budget. Quantization applies to Stage I flow-matching targets and posterior-mean anchors, and the Stage II decoder's latent inputs. Both models use the same three-task WidowX evaluation, with 24 rollouts per task.

Latent representationWidowX success rate
Continuous41.7%
Product-quantized (16-bit)30.6%

With the teacher fixed, the continuous model's success rate is 11.1 percentage points higher under this 16-bit PQ setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.