acceptodds
Under review as a conference paper at ICLR 2027

DreamX: Video Diffusion Model is a Universal Latent Action Learner

Abstract

Learning a universal action representation from video requires modeling dynamics that remain consistent across different executions of an action while being invariant to changes in embodiment and appearance. Existing latent action models (LAMs) infer actions from visual transitions, but their representations often conflate the underlying action with irrelevant visual variations, which limits transfer across embodiments. We introduce DreamX, a framework that transforms a pretrained video diffusion model into a universal latent action learner. Our approach extracts compact latent actions from the model's rich spatiotemporal features, compressing each latent video frame by 300 while using its generative prior to disentangle action-relevant dynamics from visual appearance. The resulting latent space serves as a shared action interface across human and robot embodiments, environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art representation quality. More importantly, actions encoded in one setting remain effective in another: the universal latent space supports any-to-any video transfer across embodiments and environments, as well as zero-shot cross-embodiment action transfer. This shared interface also extends beyond representation learning to control, where a compact latent policy improves task performance while lowering the computational cost of video-diffusion-based control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent action, latent action model, cross-embodiment learning, robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qEHVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DreamX repurposes a pretrained Cosmos-Predict2.5 2B video diffusion model for latent action learning. Learnable queries extract action representations from its spatiotemporal features, while a separately adapted decoder reconstructs future videos and, when labels are available, predicts robot actions. A subsequent variational bottleneck compresses the representations for policy learning without future-video generation. DreamX reports state-of-the-art LARYBench probing results, zero-shot human-to-robot video transfer, qualitative executable action transfer, and improved real-robot drawer-closing performance.

Critical strengths
  1. Strong empirical results. DreamX outperforms prior latent-action models on LARYBench and zero-shot human-to-robot video transfer.
  2. Promising real-world human-to-robot transfer. The real-robot demonstrations are particularly interesting. DreamX reproduces fine-grained human motions from a single demonstration video without paired human–robot training data. This is an impressive example of in-context imitation and a promising direction for robot learning.
Critical weaknesses
  1. The main comparisons are not apples-to-apples, making the source of the improvements unclear.
    • Unequal supervision. DreamX uses language-conditioned encoding (Eq. 2) and auxiliary ground-truth action supervision (Eq. 4; 78% of pretraining hours), while several LAM baselines rely on video-only learning. Both signals may benefit LARYBench probing. It is also unclear whether language is used during evaluation.
    • Unequal capacity and inputs. DreamX uses a pretrained 2B Video DiT with up to 32 × 2,048-dimensional latent features, while baselines differ substantially in model capacity, latent dimensions, and temporal inputs (Table 5). The probing gains may partly reflect greater representation capacity rather than better latent actions.
    • Missing controls. The video-only ablation uses a smaller training subset (Table 4). Comparisons with simple readouts from the same pretrained backbone or alternative encoders under matched supervision are missing, making it difficult to isolate the contribution of the proposed method.
  2. The attention design and action-centric representation need stronger justification.
    • Temporal causality. Although query-to-video attention is frame-causal, unrestricted query-to-query attention allows future information to reach earlier queries indirectly. This makes the temporal meaning of each latent-action token unclear.
    • Disentanglement. The original latent space is high-dimensional (up to 32 × 2,048) without explicit information-bottleneck regularization. Future-video reconstruction alone does not ensure that the latents encode actions rather than appearance or embodiment information.
    • Missing analysis. The attention design is not ablated, and appearance or embodiment leakage is not directly evaluated. The perturbation experiments (Table 8) demonstrate that latents influence generation, but do not establish that they capture transferable actions independently of other visual factors.
Questions for authors
  1. Supervision and fairness.
    • Is language provided to the encoder during LARYBench evaluation?
    • Can the authors report full-data DreamX results without language conditioning or ground-truth action supervision? How would this affect the comparison with video-only LAMs? also, since real robot actions directly supervise the latent, DreamX is semi-supervised rather than unsupervised latent action model in the usual sense, Should it then be compared with methods that also use action labels?
    • Prior work [1] shows that even limited action supervision can substantially improve downstream performance, yet DreamX's probing results barely change (Table 4). Does action supervision also have little effect on downstream control, or is probing insufficient to capture its benefits?
  2. Model capacity and pretrained priors.
    • How much of the gain comes from the encoder’s capacity (2B), as opposed to the pretrained prior, given that typical IDMs are much lighter (e.g., VPT [2], typical 0.4B)? an ablation study on encoder size, or compare with a lightweight IDM trained under the same supervision, would help clarify this.
    • How does reading latents directly from the backbone compare with using pretrained spatiotemporal features as an alignment target for a lightweight latent action model (e.g., Olaf-World [3])?
  3. Implementation details.
    • Which noise-level/timestep distribution is used to train the decoder? This loss is the signal that shapes the latent space. In addition, decoder video tokens appear to attend to all other video frames, so each frame can draw on later frames, and through them on later queries. Does this undermine the frame-causal conditioning and allow information to leak between frames? How sensitive are the results to both choices?
  4. Attention design. Why is query-to-query attention unrestricted despite the frame-causal formulation? How does causal query-to-query attention affect representation quality and transfer?
  5. Natural human-to-robot transfer. I find the real-robot demonstrations particularly impressive. However, the human demonstrations appear to involve structured motions that closely resemble robot end-effector trajectories. Does successful transfer require such robot-like demonstrations? How well does DreamX generalize to natural human manipulation, where hand gestures, grasping, and contact patterns may differ substantially from those of a robot gripper?

[1] Nikulin et al. Latent Action Learning Requires Supervision in the Presence of Distractors. 2025.

[2] Baker et al. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. NeurIPS 2022.

[3] Jiang et al. Olaf-World: Orienting Latent Actions for Video World Modeling. ICML 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer qEHV (1/3)

We thank the reviewer for the constructive feedback and for recognizing the motivation and the strong results of our method. We address each point below; revisions to the paper are marked in blue.

Q1. Supervision and Fairness

[Q1-1, W1-1] Language at evaluation. For every LARYBench task, we feed a fixed, content-free instruction, “An agent performs an action.” for fair comparison with other baselines. The text stream carries no task information at evaluation. Language conditioning is used only during pretraining, where it acts as a regularizer that lets the frozen backbone stay close to its pretraining distribution.

[W1-2] Latent dimension and temporal inputs. Table 1 already includes FAE-32, which compresses the DreamX latent from 32 × 2,048 to 32 × 32 dimensions. FAE-32 still outperforms every baseline (0.22 MSE / 67.2% accuracy vs. 0.38 / 62.3% for the strongest baseline, DiLA). The gain therefore reflects what the latents encode rather than how large they are. Also for temporal input, video-based models are evaluated on exactly the same number of frames.

[Q1-2] Effect of action supervision. We run the ablation with and without action supervision on the same reduced subset, since training on the full corpus was not feasible within our compute budget. Removing action supervision changes the results only marginally, from 0.23 / 70.49% to 0.24 / 70.02%, and the RGB-only variant still outperforms all video-only baselines in Table 1 (0.24 / 70.0% vs. 0.38 / 62.3% for DiLA). The latent actions are therefore shaped mainly by the pretrained video prior and generative decoding rather than by action labels.

As the reviewer notes, the full DreamX model uses action labels when they are available. These labels serve only as optional auxiliary supervision (Sec. 3.2, Eq. 4), and the RGB-only variant above is fully unsupervised. For comparison with methods that use action labels, Table 1 already includes villa-X, which uses action labels during training, and DreamX outperforms it by a large margin (0.17 vs. 0.87 MSE, 69.7% vs. 20.9% accuracy).

supervisionMSE ↓Acc ↑
RGB only0.2470.02
RGB and action0.2370.49

[Q1-3, W2-2] Action supervision under distractors. We thank the reviewer for pointing to this work. We perform video transfer from source videos with dynamic backgrounds, using models trained with and without action supervision (Appendix E.5, Figure 25). In both cases, the agent’s motion is reproduced in the target scene, while the background dynamics are not carried over. Similarly, although most of LARYBench consists of controlled setups, AgiBotWorld is collected with dynamic background activity, and there removing action supervision does not change regression MSE (0.32 in both cases, Table 4). Unlike the scratch LAMs studied in [1], our encoder reads from a pretrained video prior, which already encodes motion-relevant information without action labels. Action supervision therefore acts mainly as a mild regularizer and has limited effect on the learned latents.

AuthorsOpenReview ↗

Response to Reviewer qEHV (2/3)

Q2. Model capacity and pretrained priors

[Q2-1, W1-2] Encoder’s capacity and prior comparison. We run a controlled ablation that separates the two factors. We train two randomly initialized encoders, a 2B encoder with the DreamX architecture and a 0.4B encoder with fewer blocks, using the same supervision and data.

EncoderMSE ↓Acc ↑
Pretrained (DreamX)0.2370.49
Scratch 2B0.6548.30
Scratch 0.4B0.6150.7

Scaling the encoder from 0.4B to 2B without the pretrained prior yields largely similar results, whereas initializing the same 2B encoder from the pretrained backbone improves them from 0.65 to 0.23 MSE and from 48.3% to 70.5% accuracy. The gain therefore comes primarily from the pretrained prior rather than from capacity.

[Q2-2, W1-3] Reading from backbone vs. future alignment. To compare the two ways of using a pretrained model’s representation, we report LARYBench results of three variants that share the same V-JEPA 2 [2] prior: a naive latent action model (Olaf-LAM), the same model trained with alignment to future V-JEPA 2 features as in Olaf-World [3] (Olaf-Aligned), and a direct readout of the V-JEPA 2 features (V-JEPA 2):

EncoderMSE ↓Acc ↑
Olaf-LAM0.6654.43
Olaf-Aligned0.5656.3
V-JEPA 20.2166.23

Feature alignment improves the latent action model from 0.66 / 54.4% to 0.56 / 56.3%, whereas reading out the same features further improves to 0.21 / 66.2%. This suggests that reading out the pretrained features directly preserves more action-relevant information than using them only as an alignment target.

[Q3] Implementation details.

We train the decoder with a noise distribution that favors high noise levels, where the structure and motion of a video are largely determined [5]; at these levels, the decoder must rely on the latent actions to reconstruct the motion. Specifically, we sample u∼U(0,1)u \sim \mathcal{U}(0,1) per clip and set σ=5u/(1+4u)\sigma = 5u/(1+4u) (timestep 1000σ1000\sigma). We included this detail to Appendix B.2.

To examine whether later query groups affect earlier frames through the bidirectional attention of decoder video tokens, which we keep to preserve the pretrained video distribution, we replace one query group (Q2) with the same-index group from another video (Appendix D.4, Figure 16). This changes the earlier RGB frames 1–4 by only 0.0080 L1, versus 0.0254 when all groups are replaced, so this backward influence is present but small.

AuthorsOpenReview ↗

Response to Reviewer qEHV (3/3)

[Q4, W2-1] Attention design

We ablate this design by making query-to-query attention causal while keeping all other settings fixed (Table 4). Bidirectional attention performs better, reducing regression MSE from 0.35 to 0.23 and increasing classification accuracy from 66.73% to 70.49%.

Since the encoder acts as an inverse dynamics model that infers latent actions from a fully observed clip, non-causal inference is natural, as in VPT [4]. Query-to-video attention remains frame-causal, so each query reads visual evidence only to its own transition, while query-to-query attention shares clip-level context that helps resolve ambiguous transitions, such as small or occluded motions.

Bidirectional attention also yields smoother video transfer (Appendix D.3, Figure 15), and each query group retains its temporal role, as replacing a later group shifts the strongest changes to later decoded frames (Appendix D.4, Figure 16).

Query-to-query attentionMSE ↓Acc ↑
Bidirectional (DreamX)0.2370.49
Causal0.3566.73

[Q5] Natural human to robot transfer

We provide additional action transfer with natural human demonstration as a source video (Figure 6, panel (4)). However, accurate transfer remains limited for fine-grained manipulation, such as grasping small objects, where differences between the human hand and the robot gripper affect how contact is made. We added a discussion of this limitation and directions for future work in Appendix A.

[1] Latent Action Learning Requires Supervision in the Presence of Distractors, ICML 2025.

[2] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, Arxiv 2025

[3] Olaf-World: Orienting Latent Actions for Video World Modeling, ICML 2026

[4] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos, NeurIPS 2022

[5] MotionClone: Training-Free Motion Cloning for Controllable Video Generation, ICLR 2025

Reviewer_1AN6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents DreamX, a framework that learns transferable latent action representations from pretrained video diffusion models. The authors use learnable query tokens to extract motion information from the model's spatiotemporal features, and train a decoder to reconstruct future videos conditioned on these latent actions. The learned representation supports video transfer across different scenes and embodiments, as well as human-to-robot action transfer without paired training data. The authors further develop a compact latent-action policy that predicts action representations without generating future videos.

Critical strengths

First, the paper addresses an important limitation of existing latent action models, where learned representations may capture appearance changes rather than the underlying motion. Using a pretrained video diffusion model to extract action-relevant information is a natural and interesting direction. I particularly appreciate the attempt to make use of both the representation and generation capabilities of the same pretrained model.

Second I think the experimental results across several settings are strong. DreamX achieves clear improvements over existing latent action models on LARYBench, especially in action regression. The zero-shot human-to-robot video transfer results are also impressive, with consistent improvements across all reported metrics. These results provide good evidence that the learned representations capture useful motion information.

Interesting cross-embodiment transfer capabilities is alos meritorious. One of the most appealing aspects of this work is that latent actions extracted from human videos can be transferred to robot embodiments without paired human-robot training data. The qualitative results demonstrate several interesting behaviors, including trajectory following and bimanual manipulation. This is a useful step toward learning robot behaviors from larger collections of human videos.

Critical weaknesses

I think the robustness of the latent actions to changes in viewpoint is not fully evaluated. The paper aims to separate underlying motion from visual appearance and embodiment. While the cross-embodiment transfer results are encouraging, it remains unclear whether the same physical action produces consistent latent representations under different camera viewpoints. This is particularly relevant given prior work on view-invariant latent action learning, such as MuCoLA (ICML 2026). A direct evaluation of viewpoint changes would help support the paper's claims about transferable action representations.

Also the contribution of pretrained video representations could be better isolated. A central claim of the paper is that pretrained video diffusion models provide stronger action representations than conventional latent action encoders. However, the current experiments do not fully separate the benefits of video pretraining from those of the proposed query-based architecture and reconstruction objective. A more focused ablation would help explain which part contributes most to the improvements.

Questions for authors
  1. Could the authors provide a more direct ablation of the pretrained video diffusion backbone? For example, how much performance is retained when using a separate latent action encoder or removing some of the pretrained components?

  2. How consistent are DreamX's latent actions when the same physical motion is observed from different camera viewpoints? Would the authors expect explicit multi-view consistency supervision to provide additional benefits?

  3. For the real-world policy experiments, could the authors clarify how much of the improvement comes from the learned latent actions, rather than differences in backbone pretraining or policy architecture? An additional task or a more controlled comparison would help.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1AN6 (1/2)

We thank the reviewer for the constructive feedback and for recognizing the motivation and the strong results of our method. We address each point below; all new results are added to the revised paper and marked in blue.

[W1, Q2] Viewpoint robustness

We add cross-view video transfer examples to further validate our method’s viewpoint invariance, which is shown in Appendix E.5 (Figure 22). Latent actions are extracted from one camera and decoded from the initial frame of a different camera. The examples cover two cameras of the same episode and a camera from a different scene and task.

The generated video roughly follows the source arm motion while keeping the target viewpoint and scene, suggesting that the latent actions retain motion information that transfers across viewpoints even without cross-view training. These are illustrative examples rather than a systematic evaluation, and transfer quality varies across view pairs.

Discussion. These qualitative results indicate a degree of viewpoint invariance in the learned latent actions. We conjecture that this property stems from the generative prior of the decoder: given the initial frame of the target view, the decoder already supplies the scene semantics including viewpoint and appearance, so the latent actions need to carry little view-specific information. This invariance could be further strengthened with explicit multi-view supervision as in MuCoLA [1], which DreamX supports without any architectural change by encoding one synchronized view and decoding another. We included this to future work in Appendix A.

AuthorsOpenReview ↗

Response to Reviewer 1AN6 (2/2)

[W2, Q1] Contribution of each component

Each ablation below changes one component of DreamX at a time while keeping the architecture, supervision, data subset of Table 4, and training budget fixed, and reports average LARYBench regression MSE and classification accuracy.

Pretrained video prior. We train the same 2B encoder from scratch, and additionally a separate lightweight encoder with the Cosmos-Predict2.5 architecture reduced to 4 blocks, about 0.4B parameters, also from scratch.

EncoderMSE ↓Acc ↑
Pretrained (DreamX)0.2370.49
Scratch 2B0.6548.3
Scratch 0.4B0.6150.7

Without the pretrained prior, performance drops substantially for the 2B encoder (0.23 → 0.65 MSE, 70.5% → 48.3% accuracy), and the 0.4B encoder even performs slightly better than the scratch 2B, so the gain comes from the pretrained spatiotemporal prior rather than the architecture. The query-to-video attention visualization in Figure 8 further shows that the queries of the pretrained encoder attend to the acting hand or gripper, while those of the scratch encoder spread over the background.

Generative reconstruction. We replace the generative decoder with a reconstruction decoder trained from scratch, using either a regression loss or a flow-matching loss at the maximum noise level.

DecoderMSE ↓Acc ↑
Generative (DreamX)0.2370.49
Reconstruction0.2669.76
Flow matching0.2569.55

Both reconstruction decoders underperform DreamX, and the drop is concentrated in action regression. Without a generative prior, the latent must carry most of the future-frame content, which dilutes the fine-grained motion information that regression relies on.

Query design. We make query-to-query attention causal, with all other settings unchanged.

Query-to-query attentionMSE ↓Acc ↑
Bidirectional (DreamX)0.2370.49
Causal0.3566.73

Causal attention lowers performance from 0.23 / 70.49 to 0.35 / 66.73. Shared clip-level context resolves transitions that are ambiguous in isolation and yields temporally smoother transfer (Appendix D.3 Figure 15).

[Q3] Effect of latent actions on the policy

Our real-world policy evaluation was designed primarily to test whether DreamX can learn a policy from both robot and human videos. When the robot demonstrations of the upper-drawer configuration are replaced with human videos, the policy still completes this configuration in all trials (Appendix D.1, Table 6). To examine whether the latent actions also serve as an interface for a general policy across diverse tasks, we additionally evaluate on CALVIN, comparing the DreamX policy with a variant whose action head does not leverage the generated latent actions, keeping the backbone and training otherwise identical (Appendix D.5). Conditioning on latent actions improves success at every sequence length and raises the average length from 2.378 to 2.643, with the gap widening beyond the first task: +2.3 points at ≥1 versus +5.3 to +7.0 points at ≥2 to ≥5. Since both variants share the same backbone and policy architecture, this comparison isolates the contribution of latent actions to action prediction.

Policy≥1≥2≥3≥4≥5Avg. Len. ↑
DreamX policy w/o latent actions83.461.742.529.820.42.378
DreamX policy85.767.049.536.425.72.643

[1] Multi-view Consistent Latent Action Learning for World Modeling and Control, ICML 2026

Thanks for your rebuttal! Most of my concerns have been resolved, therefore I remain my current rating.

Thank you for your response and for your positive assessment of our work. We sincerely appreciate your time and effort, and your constructive feedback has helped us strengthen the paper. We will incorporate the new results and clarifications into the final version.

Reviewer_nLhzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DreamX adapts a pretrained video diffusion transformer into a shared latent-action representation for human and robot videos. Learnable queries extract compact representations from spatiotemporal features, while initial-frame-conditioned video reconstruction and optional robot-action supervision encourage them to capture transferable dynamics. These representations support video transfer, human-to-robot action decoding, and a policy that predicts compressed action codes instead of future video.

Validation includes LARYBench probing, human-to-robot video transfer over 1,008 clips and 21 tasks, qualitative physical action transfer, and real-world drawer closing. DreamX improves average regression MSE from 0.38 to 0.17 and classification accuracy from 62.25% to 69.67% over DiLA. Its drawer policy achieves 96.7% subgoal success versus 50.0% for Mimic-Video, with 19.1× lower inference latency.

Critical strengths

Strong quantitative representation and transfer results. DreamX consistently improves regression and classification across the reported LARYBench datasets and outperforms both video-transfer baselines on all four metrics. The breadth and magnitude of these gains provide consequential evidence of a useful representation.

A shared representation connects video learning to executable control. Human-to-robot transfer without paired training data, together with a compact-code policy, makes the contribution practically meaningful. Avoiding future-video generation yields a substantial measured latency reduction while retaining useful control behavior.

Critical weaknesses

The experiments do not isolate the proposed mechanism. The main ablation removes auxiliary action supervision, but does not separately test pretrained encoder features, generative supervision, or the query architecture. Without matched controls for backbone, data, and training, the gains cannot be confidently attributed to the proposed combination of spatiotemporal and generative priors. Perturbation and attention analyses demonstrate motion sensitivity more directly than appearance invariance.

Physical validation is too limited for the breadth of the claims. Action transfer is qualitative, while policy evaluation covers one drawer-closing task family on one platform against one baseline. Trial counts and uncertainty are unreported, and the policies have different pretraining histories. The headline success metric averages subgoals rather than measuring complete-task success. These limitations weaken conclusions about general control superiority and universal action transfer.

Questions for authors

Can matched-backbone/data ablations distinguish the contributions of pretrained encoder features and generative reconstruction supervision? These would materially affect attribution of the gains. What are the policy trial counts, full-task success rates, and latency measurement conditions? How comparable are the methods’ pretraining data and compute? What exact temporal windows and language conditioning are used for each LARYBench task, and how is overlap between DreamX pretraining and evaluation data excluded?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nLhz (1/2)

We thank the reviewer for the constructive review and for recognizing our strong representation and transfer results and the practical value of the shared latent-action interface. We address each point below; all new results are added to the revised paper and marked in blue.

[Q1, W1] Attribution of the gains

Each ablation below changes one component of DreamX at a time while keeping the architecture, supervision, data subset of Table 4, and training budget fixed, and reports average LARYBench regression MSE and classification accuracy.

Pretrained encoder features. We train the same 2B encoder from scratch instead of initializing it from the pretrained backbone.

EncoderMSE ↓Acc ↑
Pretrained (Ours)0.2370.49
Scratch0.6548.3

Training from scratch drops performance from 0.23 / 70.49 to 0.65 / 48.3, so the gain comes from the pretrained spatiotemporal prior rather than the architecture. The query-to-video attention visualization in Figure 8 further shows that the queries of the pretrained encoder attend to the acting hand or gripper, while those of the scratch encoder spread over the background.

Generative reconstruction. We replace the generative decoder with a reconstruction decoder trained from scratch, using either a regression loss or a flow-matching loss at the maximum noise level.

Training ObjectiveMSE ↓Acc ↑
DreamX0.2370.49
Reconstruction0.2669.76
Flow matching0.2569.55

Both reconstruction decoders underperform DreamX, and the drop is concentrated in action regression. Without a generative prior, the latent must carry most of the future-frame content, which dilutes the fine-grained motion information that regression relies on.

Query design. We make query-to-query attention causal, with all other settings unchanged.

Query-to-query attentionMSE ↓Acc ↑
Bidirectional (Ours)0.2370.49
Causal0.3566.73

Causal attention lowers performance from 0.23 / 70.49 to 0.35 / 66.73. Shared clip-level context resolves transitions that are ambiguous in isolation and yields temporally smoother transfer in Appendix D.3 (Figure 15).

[W1] Appearance invariance

We test appearance sensitivity through video transfer: if the latents encoded source appearance or background dynamics, these would reappear when the latents are decoded in a different target scene. We transfer source videos recorded in front of a dynamic background and source videos with one or three synthetic moving distractors in Appendix E.5 (Figures 23 and 24).

The generated videos reproduce only the agent's motion, and neither the background dynamics nor the distractors appear in the target scene. The same holds in Figures 4, 5, and 17, where the target keeps its own appearance and embodiment, whereas the baselines often leak the source human hand into the robot video (Figure 5).

[Q2] Policy evaluation details

Each method is run for 5 trials per drawer configuration, 15 trials in total. A trial counts as a full-task success only if the robot fully closes the required drawers, which corresponds to the Push column in Appendix D.1. Latency is the wall-clock time from sending one observation to receiving one action chunk, measured on the single RTX 3090 GPU used in the real-robot experiments.

MethodUpperLowerBothFull-task SR ↑
Mimic-Video1/50/50/56.7%
DreamX4/55/55/593.3%

[Q3] Pretraining data and compute

As detailed in Appendix B.1, DreamX trained on 2.19k hours of human and robot video from six public datasets, using two NVIDIA B200 GPUs for 6.7 days.

AuthorsOpenReview ↗

Response to Reviewer nLhz (2/2)

[Q4] LARYBench details

All video-based models, including DreamX, use the same LARYBench temporal window, and DreamX uses a fixed, content-free prompt, “An agent performs an action.”, without any task information. For AgiBot, we use only one-third of the full dataset and exclude episodes included in LARYBench. For temporal windows, all video-based baselines receive same frames. Regression clips span 5-22 frames; for classification, the atomic robot track uses 2 frames and the composite tracks use 9-frame clips.

[W2] Additional policy evaluation

Action transfer. Beyond the structured motions in the original submission, we add a natural pick-and-place demonstration to our action transfer experiments in Figure 6, panel (4). The robot reproduces the reaching, grasping, transport, and release of the human demonstration despite the morphological gap between the human hand and the robot gripper, showing that the transfer extends to grasping and placing objects.

Policy learning. Our real-world policy evaluation was designed primarily to test whether DreamX can learn a policy from both robot and human videos. When the robot demonstrations of the upper-drawer configuration are replaced with human videos, the policy still completes this configuration in all trials (Appendix Table 6). To examine whether the latent actions also serve as an interface for a general policy across diverse tasks, we additionally evaluate on CALVIN, comparing the DreamX policy with a variant whose action head does not leverage the generated latent actions, keeping the backbone and training otherwise identical (Appendix D.5). Conditioning on latent actions increases the average completed sequence length from 2.378 to 2.643 and improves success at every sequence length. These results indicate that our latent actions function as an interface for robot control beyond a single task.

PolicyAvg. Len. ↑
DreamX policy w/o latent actions2.378
DreamX policy2.643

Thank you again for your time and consideration.

As the discussion period draws to a close, we would like to follow up on our response to your review. We would be grateful if you could let us know whether our response addresses your concerns. We would also be happy to provide further clarification or discuss any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers found our representation and transfer results strong (nLhz, 1AN6, qEHV), appreciated using both the representation and generation capabilities of a pretrained video model (1AN6), and found human-to-robot transfer without paired data practically meaningful and promising (nLhz, 1AN6, qEHV). We summarize the main additions in the revised paper below; all changes are marked in blue, and we address each question in the individual responses.

AdditionLocationReviewers
Component ablations: encoder prior (2B and 0.4B encoders trained from scratch), generative prior (reconstruction decoders), action supervision, and query attentionTable 4, Sec. 4.5nLhz, 1AN6, qEHV
Query-to-video attention of the pretrained and scratch encodersFigure 8nLhz, 1AN6, qEHV
Robustness of video transfer to viewpoint, visual distractors, and dynamic backgroundsAppendix E.5nLhz, 1AN6, qEHV
Video transfer with and without action supervision under background motionAppendix E.5qEHV
Natural pick-and-place demonstration in human-to-robot action transferFigure 6 (4)nLhz, qEHV
Policy learning from human videos without action labelsTable 3, Sec. 4.4nLhz, 1AN6
CALVIN policy ablation with and without latent actionsAppendix D.5nLhz, 1AN6
Policy evaluation details: trial counts, full-task success, and latency conditionsAppendix C.3nLhz
Temporal role of queries: query-to-query attention and query-group replacementAppendix D.3, D.4qEHV
Decoder noise schedule and LARYBench evaluation detailsAppendix B.2, C.4nLhz, qEHV
Direct readout vs. feature alignment on V-JEPA 2Response to qEHVqEHV

As all reviewers requested controlled ablations of DreamX, we provide the full per-task LARYBench results below, and we have updated the ablation table (Table 4) in the revised paper accordingly.

ComponentVariantCALVIN ↓VLABench ↓RoboCOIN ↓AgiBotWorld ↓Reg. Avg. ↓Atomic Robot ↑Composite Human ↑Composite Robot ↑Cls. Avg. ↑
—DreamX0.210.080.310.320.2370.9572.5367.9970.49
Action supervisionw/o action supervision (RGB only)0.220.090.330.320.2469.2972.8167.9770.02
Encoder priorScratch encoder (2B)0.670.440.630.860.6523.7659.4161.7248.30
Encoder priorScratch encoder (0.4B)0.630.380.590.830.6127.2361.3063.4750.67
Decoder priorReconstruction decoder (regression)0.210.080.340.390.2667.9973.1268.1969.76
Decoder priorReconstruction decoder (flow matching)0.200.080.340.380.2568.4972.5867.5769.55
Query attentionCausal query-to-query attention0.270.130.460.520.3561.7071.5066.9766.73

We also updated the teaser figure, added end-effector trajectories to the qualitative transfer figures for better visibility, and improved the attention visualizations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.