- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DreamX repurposes a pretrained Cosmos-Predict2.5 2B video diffusion model for latent action learning. Learnable queries extract action representations from its spatiotemporal features, while a separately adapted decoder reconstructs future videos and, when labels are available, predicts robot actions. A subsequent variational bottleneck compresses the representations for policy learning without future-video generation. DreamX reports state-of-the-art LARYBench probing results, zero-shot human-to-robot video transfer, qualitative executable action transfer, and improved real-robot drawer-closing performance.
Critical strengths
- Strong empirical results. DreamX outperforms prior latent-action models on LARYBench and zero-shot human-to-robot video transfer.
- Promising real-world human-to-robot transfer. The real-robot demonstrations are particularly interesting. DreamX reproduces fine-grained human motions from a single demonstration video without paired human–robot training data. This is an impressive example of in-context imitation and a promising direction for robot learning.
Critical weaknesses
- The main comparisons are not apples-to-apples, making the source of the improvements unclear.
- Unequal supervision. DreamX uses language-conditioned encoding (Eq. 2) and auxiliary ground-truth action supervision (Eq. 4; 78% of pretraining hours), while several LAM baselines rely on video-only learning. Both signals may benefit LARYBench probing. It is also unclear whether language is used during evaluation.
- Unequal capacity and inputs. DreamX uses a pretrained 2B Video DiT with up to 32 × 2,048-dimensional latent features, while baselines differ substantially in model capacity, latent dimensions, and temporal inputs (Table 5). The probing gains may partly reflect greater representation capacity rather than better latent actions.
- Missing controls. The video-only ablation uses a smaller training subset (Table 4). Comparisons with simple readouts from the same pretrained backbone or alternative encoders under matched supervision are missing, making it difficult to isolate the contribution of the proposed method.
- The attention design and action-centric representation need stronger justification.
- Temporal causality. Although query-to-video attention is frame-causal, unrestricted query-to-query attention allows future information to reach earlier queries indirectly. This makes the temporal meaning of each latent-action token unclear.
- Disentanglement. The original latent space is high-dimensional (up to 32 × 2,048) without explicit information-bottleneck regularization. Future-video reconstruction alone does not ensure that the latents encode actions rather than appearance or embodiment information.
- Missing analysis. The attention design is not ablated, and appearance or embodiment leakage is not directly evaluated. The perturbation experiments (Table 8) demonstrate that latents influence generation, but do not establish that they capture transferable actions independently of other visual factors.
Questions for authors
- Supervision and fairness.
- Is language provided to the encoder during LARYBench evaluation?
- Can the authors report full-data DreamX results without language conditioning or ground-truth action supervision? How would this affect the comparison with video-only LAMs? also, since real robot actions directly supervise the latent, DreamX is semi-supervised rather than unsupervised latent action model in the usual sense, Should it then be compared with methods that also use action labels?
- Prior work [1] shows that even limited action supervision can substantially improve downstream performance, yet DreamX's probing results barely change (Table 4). Does action supervision also have little effect on downstream control, or is probing insufficient to capture its benefits?
- Model capacity and pretrained priors.
- How much of the gain comes from the encoder’s capacity (2B), as opposed to the pretrained prior, given that typical IDMs are much lighter (e.g., VPT [2], typical 0.4B)? an ablation study on encoder size, or compare with a lightweight IDM trained under the same supervision, would help clarify this.
- How does reading latents directly from the backbone compare with using pretrained spatiotemporal features as an alignment target for a lightweight latent action model (e.g., Olaf-World [3])?
- Implementation details.
- Which noise-level/timestep distribution is used to train the decoder? This loss is the signal that shapes the latent space. In addition, decoder video tokens appear to attend to all other video frames, so each frame can draw on later frames, and through them on later queries. Does this undermine the frame-causal conditioning and allow information to leak between frames? How sensitive are the results to both choices?
- Attention design. Why is query-to-query attention unrestricted despite the frame-causal formulation? How does causal query-to-query attention affect representation quality and transfer?
- Natural human-to-robot transfer. I find the real-robot demonstrations particularly impressive. However, the human demonstrations appear to involve structured motions that closely resemble robot end-effector trajectories. Does successful transfer require such robot-like demonstrations? How well does DreamX generalize to natural human manipulation, where hand gestures, grasping, and contact patterns may differ substantially from those of a robot gripper?
[1] Nikulin et al. Latent Action Learning Requires Supervision in the Presence of Distractors. 2025.
[2] Baker et al. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. NeurIPS 2022.
[3] Jiang et al. Olaf-World: Orienting Latent Actions for Video World Modeling. ICML 2026.
5 comments
Response to Reviewer qEHV (2/3)
Q2. Model capacity and pretrained priors
[Q2-1, W1-2] Encoder’s capacity and prior comparison. We run a controlled ablation that separates the two factors. We train two randomly initialized encoders, a 2B encoder with the DreamX architecture and a 0.4B encoder with fewer blocks, using the same supervision and data.
| Encoder | MSE ↓ | Acc ↑ |
|---|---|---|
| Pretrained (DreamX) | 0.23 | 70.49 |
| Scratch 2B | 0.65 | 48.30 |
| Scratch 0.4B | 0.61 | 50.7 |
Scaling the encoder from 0.4B to 2B without the pretrained prior yields largely similar results, whereas initializing the same 2B encoder from the pretrained backbone improves them from 0.65 to 0.23 MSE and from 48.3% to 70.5% accuracy. The gain therefore comes primarily from the pretrained prior rather than from capacity.
[Q2-2, W1-3] Reading from backbone vs. future alignment. To compare the two ways of using a pretrained model’s representation, we report LARYBench results of three variants that share the same V-JEPA 2 [2] prior: a naive latent action model (Olaf-LAM), the same model trained with alignment to future V-JEPA 2 features as in Olaf-World [3] (Olaf-Aligned), and a direct readout of the V-JEPA 2 features (V-JEPA 2):
| Encoder | MSE ↓ | Acc ↑ |
|---|---|---|
| Olaf-LAM | 0.66 | 54.43 |
| Olaf-Aligned | 0.56 | 56.3 |
| V-JEPA 2 | 0.21 | 66.23 |
Feature alignment improves the latent action model from 0.66 / 54.4% to 0.56 / 56.3%, whereas reading out the same features further improves to 0.21 / 66.2%. This suggests that reading out the pretrained features directly preserves more action-relevant information than using them only as an alignment target.
[Q3] Implementation details.
We train the decoder with a noise distribution that favors high noise levels, where the structure and motion of a video are largely determined [5]; at these levels, the decoder must rely on the latent actions to reconstruct the motion. Specifically, we sample per clip and set (timestep ). We included this detail to Appendix B.2.
To examine whether later query groups affect earlier frames through the bidirectional attention of decoder video tokens, which we keep to preserve the pretrained video distribution, we replace one query group (Q2) with the same-index group from another video (Appendix D.4, Figure 16). This changes the earlier RGB frames 1–4 by only 0.0080 L1, versus 0.0254 when all groups are replaced, so this backward influence is present but small.
Response to Reviewer qEHV (3/3)
[Q4, W2-1] Attention design
We ablate this design by making query-to-query attention causal while keeping all other settings fixed (Table 4). Bidirectional attention performs better, reducing regression MSE from 0.35 to 0.23 and increasing classification accuracy from 66.73% to 70.49%.
Since the encoder acts as an inverse dynamics model that infers latent actions from a fully observed clip, non-causal inference is natural, as in VPT [4]. Query-to-video attention remains frame-causal, so each query reads visual evidence only to its own transition, while query-to-query attention shares clip-level context that helps resolve ambiguous transitions, such as small or occluded motions.
Bidirectional attention also yields smoother video transfer (Appendix D.3, Figure 15), and each query group retains its temporal role, as replacing a later group shifts the strongest changes to later decoded frames (Appendix D.4, Figure 16).
| Query-to-query attention | MSE ↓ | Acc ↑ |
|---|---|---|
| Bidirectional (DreamX) | 0.23 | 70.49 |
| Causal | 0.35 | 66.73 |
[Q5] Natural human to robot transfer
We provide additional action transfer with natural human demonstration as a source video (Figure 6, panel (4)). However, accurate transfer remains limited for fine-grained manipulation, such as grasping small objects, where differences between the human hand and the robot gripper affect how contact is made. We added a discussion of this limitation and directions for future work in Appendix A.
[1] Latent Action Learning Requires Supervision in the Presence of Distractors, ICML 2025.
[2] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, Arxiv 2025
[3] Olaf-World: Orienting Latent Actions for Video World Modeling, ICML 2026
[4] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos, NeurIPS 2022
[5] MotionClone: Training-Free Motion Cloning for Controllable Video Generation, ICLR 2025
Response to Reviewer qEHV (1/3)
We thank the reviewer for the constructive feedback and for recognizing the motivation and the strong results of our method. We address each point below; revisions to the paper are marked in blue.
Q1. Supervision and Fairness
[Q1-1, W1-1] Language at evaluation. For every LARYBench task, we feed a fixed, content-free instruction, “An agent performs an action.” for fair comparison with other baselines. The text stream carries no task information at evaluation. Language conditioning is used only during pretraining, where it acts as a regularizer that lets the frozen backbone stay close to its pretraining distribution.
[W1-2] Latent dimension and temporal inputs. Table 1 already includes FAE-32, which compresses the DreamX latent from 32 × 2,048 to 32 × 32 dimensions. FAE-32 still outperforms every baseline (0.22 MSE / 67.2% accuracy vs. 0.38 / 62.3% for the strongest baseline, DiLA). The gain therefore reflects what the latents encode rather than how large they are. Also for temporal input, video-based models are evaluated on exactly the same number of frames.
[Q1-2] Effect of action supervision. We run the ablation with and without action supervision on the same reduced subset, since training on the full corpus was not feasible within our compute budget. Removing action supervision changes the results only marginally, from 0.23 / 70.49% to 0.24 / 70.02%, and the RGB-only variant still outperforms all video-only baselines in Table 1 (0.24 / 70.0% vs. 0.38 / 62.3% for DiLA). The latent actions are therefore shaped mainly by the pretrained video prior and generative decoding rather than by action labels.
As the reviewer notes, the full DreamX model uses action labels when they are available. These labels serve only as optional auxiliary supervision (Sec. 3.2, Eq. 4), and the RGB-only variant above is fully unsupervised. For comparison with methods that use action labels, Table 1 already includes villa-X, which uses action labels during training, and DreamX outperforms it by a large margin (0.17 vs. 0.87 MSE, 69.7% vs. 20.9% accuracy).
[Q1-3, W2-2] Action supervision under distractors. We thank the reviewer for pointing to this work. We perform video transfer from source videos with dynamic backgrounds, using models trained with and without action supervision (Appendix E.5, Figure 25). In both cases, the agent’s motion is reproduced in the target scene, while the background dynamics are not carried over. Similarly, although most of LARYBench consists of controlled setups, AgiBotWorld is collected with dynamic background activity, and there removing action supervision does not change regression MSE (0.32 in both cases, Table 4). Unlike the scratch LAMs studied in [1], our encoder reads from a pretrained video prior, which already encodes motion-relevant information without action labels. Action supervision therefore acts mainly as a mild regularizer and has limited effect on the learned latents.