acceptodds
Under review as a conference paper at ICLR 2027

One from Infinity: Actualizing Futures from Pretrained World Models into Robot Actions

Abstract

A pretrained video world model admits many plausible futures for a scene, but a robot must realize the exact task-conditioned one. To turn world models into executable robot policies, existing methods fine-tune the heavy world model backbone using large-scale robot data and computational resources. Challenging this status quo, we argue that the expensive part has already been paid in the world model pretraining since the representation space of a video world model lays out the diverse potential futures. In this case, what remains is to select the future that accomplishes the task and to read out the actions that realize it. We formalize this task as actualization, which learns a task-conditioned selection and realization on top of a prior supplied by a frozen world model. This can be solved by a tiny actualizer model. We implement RoboActualizer with as few as 60M parameters on top of a frozen world model encoder. The actualizer is composed of two lightweight DiT experts that jointly predict future latents and actions by flow matching. The model can be trained entirely on a single GPU with 32 GB peak memory. With up to 100x fewer trainable parameters than existing WAMs and VLAs, RoboActualizer reaches great performance on simulation benchmarks including LIBERO, LIBERO-Plus, RoboTwin 2.0 and five tasks on two real-world platforms, with a low latency of 39ms that allows real-time control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Video World Models, World Action Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hfhEOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors argue that pretrained world models already encode scene dynamics and the space of possible futures. Therefore, the robot need not relearn these dynamics, but rather select the future that accomplishes the task and decode the corresponding actions. To this end, they propose RoboActualizer, a lightweight module with only 60M parameters built on top of a frozen world model encoder. Remarkably, the model can be trained on a single GPU (32 GB peak memory) while achieving decent performance.

Critical strengths
  1. The paper is well-organized and the writing is clear and logical. For example, the formalization of Eq. 3 cleanly decomposes the problem into three interpretable factors. This decomposition provides a clear conceptual framework and justifies why a tiny downstream model suffices.
  2. RoboActualizer achieves competitive results across multiple benchmarks while using few trainable parameters.
  3. RoboActualizer can be trained on a single GPU with 32 GB peak memory and achieves an economic inference latency, significantly faster than all baselines. This makes the approach genuinely practical for real-time robot control.
  4. The method is also evaluated on two real robot platforms with multiple manipulation tasks.
Critical weaknesses

The approach's performance is inherently bounded by the quality and coverage of the frozen world model's prior. While Table 7 shows that larger world models yield better results, the paper does not discuss the saturation point or what happens when the world model's pretraining distribution significantly mismatches the deployment domain. A discussion of this limitation and potential remedies (e.g., light fine-tuning of the encoder) maybe valuable.

Questions for authors

please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hfhE

We appreciate your efforts in reviewing our paper. We try to address your concerns as follows:

W1: The saturation point of world model scaling and what happens when the pretraining distribution mismatches.

Our results support the first half of the reviewer’s premise: a stronger world model encoder yields a stronger policy. We trained the same actualizer on frozen V-JEPA 2.1 encoders of three sizes. Moving from 80M to 300M improves LIBERO-Plus from 50.7% to 60.0% (Table 7), far larger than widening the actualizer to DiT-B. Moving from 300M to 2B, however, gives no noticeable further gain. We believe this reflects a ceiling of the encoders rather than our method: the 300M V-JEPA 2.1 model is distilled from the 2B model, and the two are close on the original authors’ own temporal benchmarks (DAVIS label propagation: 67.0 / 68.7 / 69.0 for 80M / 300M / 2B). In other words, the 2B encoder is not substantially stronger than the 300M one by V-JEPA’s own measure, so there is little additional signal for the actualizer to read out. Our results track this curve closely, which is consistent with our claim rather than contradicting it. Locating the true saturation point would require encoders that are stronger than the current model, and pretraining such a model is beyond our compute budget. We list it as future work. As stronger open video encoders appear, our method can adopt them.

For the distribution mismatches, V-JEPA 2.1 is pretrained on real-world internet video, while all our evaluations are on simulated robot scenes (LIBERO, LIBERO-Plus, RoboTwin) and two real robots, so the reported results already include a non-trivial pretraining–deployment gap. LIBERO-Plus in particular applies visual perturbations unseen in both pretraining and fine-tuning, and the frozen encoder transfers well there. To test whether closing this gap helps, we fine-tuned the 300M encoder on LIBERO with LoRA using the V-JEPA 2.1 objective and retrained the actualizer on top. LIBERO is unchanged (98.0%) and LIBERO-Plus drops to 60.9%. We read this as slight overfitting of the encoder to the LIBERO distribution, which argues for keeping the general-purpose frozen encoder: it generalizes better under perturbation and costs nothing on robot data.

Thanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.

Reviewer_hfhEOpenReview ↗

Response to authors rebuttal

OK,I have been positive about this work from the start. I also take a quick look at the other reviews. I'm sticking with my rating.

Thank you very much for your support, and for spending time reviewing our paper and reading other reviews. Please feel free to let us know if you have any further concerns.

Reviewer_MA9eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RoboActualizer, a V-JEPA based Fast-WAM style policy the claims to make the method cheaper, with lower latency, and strong performance compared to chosen baselines.

Critical strengths

This paper could be interpreted as a mix between world models used as planners, and world action modelling: since methods usually fall into one category or the other, this is an interesting perspective (the rest of the paper doesn't really follow that intuition since there is no actual selection or planning in the WAM, but that idea which is hinted at is promising).

Critical weaknesses

Several major issues:

  1. You seem to be claiming that there are several possible futures and that your method somehow collapses onto the ones where the task is realized. This is basically what every WAM does, given that the video backbones are trained with video and language, making the instruction-based collpase native to the method. Your method as it is implemented doesn't seem to push it further than this. Your method is basically FastWAM with a different initialization and latent space, the whole justification doesn't seem to result in anything. Moreover, even with a current state and an instruction, the number of futures, or ways to solve a task is still infinite and doesn't collapse just given the text, as you seem to say. Choosing between those futures would be the interesting part of the project, but it is not done at all here.

  2. Your theory relies on a world model, which is a predictor of the future until a given horizon, but you say that you use V-JEPA2.1 as world model. As it is an encoder and not a future predictor of any kind, this shows a crucial misunderstanding of the concept and the parts at work in the method.

  3. The baselines are heavily cherry picked as 63% is nowhere near good performance for their LIBERO-Plus headline number. For example, the performance reported for π0\pi_0-FAST actually outperforms your method with 64% in the original LIBERO-Plus paper (https://arxiv.org/html/2510.13626v1#S6.T2). If 63% wasn't near good performance when the benchmark came out, the performance has drastically increase on LIBERO-Plus, and most decent baselines from 70% to have been omitted 90%.

  4. There is no real study that would help this paper to get past the weak numbers, given that the only added explanatory experiment here is a simple pca. The DINO / WAM / V-JEPA comparison could be interesting, but given the state of the method and the inconsistencies throughout, it is hard to take anything from it.

Questions for authors

I refer you to the weaknesses.

11 comments
AuthorsOpenReview ↗

Response to Reviewer MA9e (1/2)

We appreciate your efforts in reviewing our paper. We try to address your concerns as follows:

W1: The instruction-conditioned collapse is native to all WAMs given that the video backbones are trained with video and language.

To begin with, we would like to clarify a factual point, i.e. our backbone is not pretrained with language. V-JEPA 2.1 is trained on videos alone without any languages supervision or inputs. And our lightweight actualizer is trained on top of the V-JEPA latent from scratch.

Thus, the instruction-based actualization is not native to our pretrained model but comes from the actualization learning process on robot data with little computation. This also highlights an important difference between our method and previous WAM like Fast-WAM. Previous WAMs inherit the instruction-following ability from the pretrained video generation model and need to finetune the entire huge generative model backbone with billions of parameters to adapt this ability to robot policy.

We agree the implementation contains no explicit enumeration over futures and have revised the manuscript accordingly to avoid confusion. Our key contribution is to exploit the future prediction ability inside the latent space to reduce the cost of dynamics modeling and make the policy learning cheaper. This is the reason why a lightweight DiT actualizer suffices and which allows us to outperform large models like Fast-WAM with much less computation. In Table 6, we highlight that ability for future prediction inside the latent space from pretraining and cross-frame modeling is the key for performance improvement.

It is true that there are still many ways to solve the task even if there is language instruction, and this is the reason why we use flow matching to model the conditional distribution and sample one action for execution.

W2: V-JEPA is an encoder instead of world model or future predictor.

We understand that, strictly speaking, the V-JEPA world model consists of both encoder and predictor (https://ai.meta.com/research/vjepa/) while we only use the encoder in our paper. Thus, we have rephrased the term more precisely as "world model encoder" instead of "world model" in the revised version.

We also understand the reviewer's concern that the world model encoder cannot explicitly predict the future latent. To this end, we highlight that our claim and formulation don't rely on explicit future prediction and this is a difference between our method and MPC. In Sec. 3.1, we said the world model encoder "provides a representation of the scene that already contains its future dynamics and evolutions", which is the premise of our formulation. This statement is consistent with how the V-JEPA authors claim: V-JEPA 2.1 reaches state-of-the-art Ego4D short-term anticipation with “frozen encoder” plus an attentive probe, attributed by the authors to its “predictive capabilities.” In Table 6, we also test it directly by applying the same DiT actualizer to frozen 1-frame or 4-frame DINO features (39.7% and 29.8%), which cannot get reasonable performance despite the strong representation ability in DINO features. All the results support that the future prediction ability rather than representation ability inside the latent space is crucial.

W3: Baselines for LIBERO-Plus.

We thank the reviewer for pointing out the concerns about LIBERO-Plus baselines. For the specific number, π0\pi_0-FAST result was modified to 61.6% in the newest version of LIBERO-Plus paper (https://arxiv.org/html/2510.13626v3#S6.T2). For the baseline coverage, we agree that some baselines were missed in the original manuscript (OpenVLA-OFT, NORA, RIPT-VLA). We include them from the LIBERO-Plus paper in Table 2 of the revised version. Although some methods can achieve higher performance, they are pretrained on large-scale robot data, which is also reflected in the new table. According to the LIBERO-Plus paper, their 79.5% result comes from additional finetuning on perturbed images, which is not a fair comparison with other zero-shot generalization models. Our method still achieves the best performance among methods without robot pretraining. Some concurrent works are discussed in the related work section separately. We highlight that our method is lightweight (300M for frozen V-JEPA and 60M for trainable DiTs). The 60M trainable parameters are significantly fewer than many baselines (e.g. 3.3B in π\pi-series and 6B in FastWAM). Under the same strict data and evaluation protocol as Fast-WAM — clean LIBERO demonstrations, no embodied pretraining, no augmentation toward the LIBERO-Plus distribution — RoboActualizer improves zero-shot LIBERO-Plus performance significantly with a single GPU for training compared to FastWAM (62.2% vs 50.0%) in the controlled experiment, which supports our contribution.

AuthorsOpenReview ↗

Response to Reviewer MA9e (2/2)

W4: No study or explanatory experiment except PCA.

We would like to emphasize that the paper contains multiple key ablation studies beyond PCA analysis in Fig. 3, and all of them are designed as support of our major claims and contributions. Table 5 (removing latent prediction, 60.0% drops to 17.9%) shows that determining the future latent is a critical training target aligned with our claim of task-conditioned actualization. Table 6 compares multiple representation spaces, including DINOv3, WAN VAE, and V-JEPA image vs. video, which shows that the future prediction ability in the latent space from cross-frame modeling is more important than the representation ability. This is also a key support for our introduction of world model encoder. In Table 7, we show that stronger world model encoder is more helpful than larger DiT actualizer, which aligns with our claims that the actualizer only needs to determine and realize the task-conditioned future rather than modeling the scene, and points out a scaling direction. Table 8 shows the role of some architecture details, where x-prediction makes it possible to use small actualizers to model the latent space.

Thanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.

W1–W2

I apologize that my wording about instruction collapse was not the clearest: I used “video backbone” as shorthand for the transformer in the MoT supervised on future prediction, rather than the frozen encoder.

After checking your implementation, my concern is that you retain the same functional block structure and attention dependencies as Fast-WAM. The representation and observation history differ, but the roles of the experts remain the same. The main text decomposes the quantities to learn in Eq. (3), yet I cannot identify an operation or loss that implements that decomposition rather than the existing future/action co-training objective.

Overall, the rebuttal does not resolve my main concern about the correspondence between the mathematical formulation and the blocks used in the paper and code. To be explicit, in this Fast-WAM-style policy there are:

  • An encoder: a VAE, V-JEPA or another visual encoder maps observations and futures into latent features. Here, V-JEPA encodes an observed history. It is not itself the component trained on robot demonstrations to predict future latents. The paper calls this encoder gϕg_\phi in Eq. (1).
  • A visual expert: this is the first expert, supervised on future prediction conditioned on the observed features and instruction. It learns the future distribution, rather than separately learning the determination factor in Eq. (3).
  • An action expert: this is the second trainable expert, supervised on action prediction conditioned on observed context, instruction and proprioception. It does not attend to future tokens.

After reviewing the paper and provided code further:

  • Your first expert does not separately implement determination as claimed in §3.4. It learns task-conditioned future prediction, as in Fast-WAM-style co-training.
  • The second expert does not implement realization either as mapping a selected future into actions: just as in Fast-WAM, it cannot attend to future tokens, and neither can the observation-context tokens it reads.
  • Eq. (3) describes the future prior as given by gϕg_\phi, but the implementation only uses gϕg_\phi to encode observations and futures. The distribution of future latents is learned by the trainable visual expert, not the V-JEPA model.

My concern is still that §3.1 introduces a stronger mathematical interpretation that is not followed by any corresponding implemented mechanism.

An empirical study of latent representations and compact WAM policies could certainly be a useful contribution. However, this paper is framed around a methodological improvement, selecting and realizing an existing future prior, that the implementation and experiments do not implement or establish.

It is possible that the clarity of the text has caused me to misunderstand the intended contribution. To clarify, I have the following more precise questions:

  1. What in your architecture or losses implements the determination and realization factors in Eq. (3) that the base implementation of FastWAM does not?
  2. What distinguishes “selecting an already-established future” from normally learning instruction-conditioned future prediction using ? Which experiment support these explanations, rather than simply showing that the selected recipe work better than FastWAM?
  3. How does the action expert 'realize' a selected future when it cannot access future tokens?

W3

The claim that “our method still achieves the best performance among methods without robot pretraining” is wrong and a strong overclaim. Known baselines that have been available for long enough to not be concurrent work are :

  • ImageWAM: 82.2%.
  • Cosmos Policy: 82.2%.
  • StarVLA: 74.1%.
  • Qwen-RobotManip-scratch: 78.3%.

Concurrent work adds further context:

  • LeapBot-WA:73.1% (Libero model has no robotic pretraining)
  • La-JEPA-L: 84.1%
  • NowWAM: 87.7%

Even without taking concurrent work into account, your method is 20 point under SOTA. If we do look at concurrent work, this delta increases, even when looking at highly similar methods like LeapBot-WA and La-JEPA.


Overall

All in all, the rebuttal does not resolve my earlier concerns. Unless the answers to the added questions demonstrate a misunderstanding of the work or code on my part, I will maintain my strong-reject recommendation. The central issue is not just the terminology: the claimed probabilistic mechanism is not established by the implemented model or the supported in experiments. In my view, the work needs substantial revision of its formulation, positioning and evidence to meet the conference's standards.

Reviewer_oQzQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RoboActualizer proposes a lightweight robot policy that pairs a frozen 300M V-JEPA 2.1 ViT-L encoder with a ~60M parameter Mixture-of-Transformers (MoT) diffusion head. Trained via flow matching, it jointly predicts future V-JEPA latents and 16-step action chunks based on text and state inputs. The paper's core conceptual claim is that a frozen video world model already models multi-future dynamics, meaning a downstream policy only needs to "select" the task-relevant future and "actualize" the actions (Equation 3). At inference, the future-latent branch is dropped, enabling a fast 39.9 ms execution. The model is evaluated on LIBERO, LIBERO-Plus, RoboTwin 2.0, and two real-world platforms (FANUC and Unitree G1).

Critical strengths

The paper has the following valuable aspects:

  • The method achieves an efficient and competitive baseline (98.0% on LIBERO, 58.8% on RoboTwin 2.0) with a highly accessible compute footprint. By freezing the 300M V-JEPA encoder and training only a 60M Mixture-of-Transformers (MoT) head on offline cached latents, the entire model optimizes in roughly 15 hours on a single 32GB PRO 6000 GPU (Section 4.4). This avoids the massive fine-tuning costs typical of full-parameter VLA models and translates well to data-constrained settings.

  • Architecturally, the method introduces genuine value through its lightweight MoT diffusion head and its use of clean-sample parameterization. The MoT structure elegantly separates the future "selection" and action "realization" tasks into distinct but interacting experts. Furthermore, adapting prediction for the low-rank latent space allows this narrow 60M network to succeed where standard full-rank Gaussian noise prediction might fail. The necessity of this joint architecture is cleanly validated: Table 5 demonstrates that removing the auxiliary latent prediction causes LIBERO-Plus success to collapse from 60.0% to 17.9%, proving it is a powerful mechanism for reading frozen representations.

  • Finally, the paper demonstrates exemplary transparency in its supplementary materials. Appendices B–D provide unusually thorough and reproducible details regarding data processing, exact normalization statistics, 3D RoPE spatial splits, and precise shifted rectified-flow schedule parameters. This level of granularity ensures the community can rigorously reconstruct the baseline without guessing hyperparameter values.

Critical weaknesses

However, Reviewer found the following issues:

A. Main concerns

  • In Table 2, the reported LIBERO-Plus Overall score is 63.1%. However, recalculating the unweighted mean of the seven provided sub-metrics (37.6, 51.1, 68.6, 85.1, 60.4, 54.5, 60.9) yields 59.7%. Using the official LIBERO-Plus weighted metric yields an even lower 58.5%. Both recalculations contradict the headline 63.1% and would drop the method's rank below π0-Fast and DC-WAM. Please clarify this point.

  • The paper frames the method as a novel paradigm of "selecting a future from infinity" (Eq. 3). In reality, the implementation (Eq. 6) is standard conditional flow matching on a single ground-truth demonstration target. Dropping future latents at test time to accelerate inference is directly lifted from Fast-WAM [1]. Furthermore, coupling action generation with future latent prediction on top of frozen encoders has been established by prior work such as JEPA-WAM [2], FOCA [3], and VLA-JEPA [4]. Stripped of its florid theoretical framing, the architecture's genuine novelty is extremely narrow: it is merely a smaller 60M-parameter DiT using clean-sample parameterization (x-prediction) to fit a low-rank space.

B. Other ones

  • Section 4.4 states that future latent tokens are discarded at inference to reduce latency. However, Appendix B.1 describes a "single joint self-attention over the concatenated token stream" without mentioning any block or causal masking during training. If action tokens attend to future tokens during training, dropping them at inference would cause a severe distribution shift.

  • To prove that a video world model is necessary, Table 6 compares V-JEPA to DINOv3. However, DINOv3 is restricted to a 1-frame input, while V-JEPA receives a 4-frame temporal clip. This is not an apples-to-apples comparison, confounding the benefits of the world model with the benefits of a multi-frame observation history.

  • Section 3.3 claims the V-JEPA latent space is "inherently low-dimensional" to justify a narrow DiT. Yet, Figure 3 shows that 384 dimensions are required to capture 93.8% of the variance. Given the original space is 1024D, requiring >37% of the dimensions stretches the definition of "inherently low-dimensional."

[1] Yuan, Tianyuan, et al. "Fast-wam: Do world action models need test-time future imagination?." arXiv preprint arXiv:2603.16666 (2026).

[2] JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

[3] FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation learning new manipulation skills from a handful of demonstrations, ICML 2026

[4] VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model

Questions for authors

I would like to receive feedback from the following ones:

a. Could you provide the exact formula or weighting used to calculate the 63.1% LIBERO-Plus Overall score in Table 2 from the sub-category metrics?

b. Given that dropping future latents at inference mirrors Fast-WAM and joint latent-action prediction mirrors FOCA and JEPA-WAM, what is the fundamental algorithmic novelty beyond shrinking the DiT to 60M parameters and using x-prediction? It would be convincing if the authors could include a comparison to one of these baselines in simulation (for instance, FOCA has results on Libero).

c. Where in the loss function (Eq. 6) or architecture is the explicit "selection" factor from Eq. 3 implemented, beyond standard conditional flow matching on the target demonstration?

d. During training, are action tokens masked from attending to future latent tokens? If not, how does the model handle the sudden absence of future tokens during inference?

e. In Table 6, how does a 4-frame version of DINOv3 perform? Comparing a 1-frame spatial encoder to a 4-frame spatio-temporal encoder isolates the wrong variable.

6 comments
AuthorsOpenReview ↗

Response to Reviewer oQzQ (1/2)

We appreciate your efforts in reviewing our paper. We try to address your concerns as follows:

W1: Unreproducible LIBERO-Plus Overall Score

Thank you for pointing out this discrepancy. We apologize for the transcription error in the last row of Table 2. We mistakenly transcribed the number without CFG. The correct results are provided below. Our experiments show that CFG won't hurt latency as the parallel inference of our small model is far from GPU saturation.

CameraRobotLang.LightBg.NoiseLayout
39.560.570.787.256.959.067.8

We also found that the Overall column mixed two conventions (the unweighted mean of the seven dimensions for Fast-WAM, and ours, the instance-weighted mean used by the LIBERO-Plus leaderboard for the others). The revised table reports the instance-weighted mean used by LIBERO-Plus leaderboard for every row: Fast-WAM 50.0, and RoboActualizer 62.2.

To ensure the reproducibility, we also include an anonymous link to the code and checkpoints in the response for your reference:

The code and checkpoints will also be released in the camera ready version as well.

W2: Overstated Novelty and Misleading Framing. What is the fundamental algorithmic novelty beyond shrinking the DiT to 60M parameters and using x-prediction? Where in the loss function (Eq. 6) or architecture is the explicit "selection" factor?

The key contribution of this paper lies in that we identify the latent space of world model encoder with future prediction ability can help the policy learning and enable a remarkably compact policy to learn task-conditioned behavior. This makes our paper different from other works mentioned by the reviewer (FOCA, JEPA-WAM, VLA-JEPA). As we note in Sec. 4.5.2, Other works "consider latent features as a superior alternative to RGB images due to their rich high-level semantics", so they still require to finetune a large model to learn the future prediction. In contrast, we make full use of the future prediction ability in the world model latent space, which is the key that enables to reduce the computation in robot data finetuning. Following the reviewer’s suggestion, we have added FOCA to Table 1. On LIBERO, FOCA reports 96.6% average success with 3.2B trainable parameters initialized from pretrained π0\pi_0, compared to 98.0% for RoboActualizer with 60M trainable parameters and no embodied pretraining. FOCA and VLA-JEPA simply use video latent as additional supervision for VLA training without utilizing the future predictive ability in the latent, and they have to finetune a large VLA backbone to model the dynamics. JEPA-WAM is discussed in Sec. 2. The main design difference from JEPA-WAM, single-frame (strong representation w/o future prediction ability) versus clip-level encoding, is tested under controlled conditions in Table 6 (46.4% vs. 60.0%). This is also the reason why other works all require finetuning larger models or use more data and computation.

For the term "selection", we consider it as an implicit determination rather than explicitly rolling out all the possible futures. This is also the key difference between our method and existing MPC-based algorithms. Following the suggestions of reviewers, we have revised our formulation in Eq. 3 consistent with our MoT architecture: qθ(zt:t+H+,at:t+H∣zt,ℓ)=qθ(zt:t+H+∣ht,ℓ)⏟determination: fits p(zt:t+H+∣zt,ℓ) in Eq. (2)⋅qθ(at:t+H∣ht,ℓ)⏟realization: action prediction,ht=fθ(zt,ℓ)q_\theta(\mathbf z^+_{t:t+H},\mathbf a_{t:t+H}\mid z_t,\ell)=\underbrace{q_\theta(\mathbf z^+_{t:t+H}\mid h_t,\ell)}_{\textbf{determination}\text{: fits } p(\mathbf z^+_{t:t+H}\mid z_t,\ell)\text{ in Eq. (2)}}\cdot\underbrace{q_\theta(\mathbf a_{t:t+H}\mid h_t,\ell)}_{\textbf{realization}\text{: action prediction}},\quad h_t=f_\theta(z_t,\ell) Currently, the two items correspond to the latent and action expert separately, and hth_t is the intermediate feature for the clean frame from latent expert. The latent expert fits one component from Eq. 2. The action expert corresponds to the action prediction. The overall process is: 1) The latent from world model encoder ztz_t includes the representation of potential future evolution; 2) The latent DiT learns to fit a specific component based on the task instruction from the prior mixture distribution, where a task-conditioned intermediate feature indicates the determined future. 3) The action DiT attends to the intermediate features of latent DiT and outputs the actions. The original term "selection" refers to step 2) that fits one task-conditioned component from the prior mixture. Given potential confusion with explicit selections through multiple rollouts, we have changed some description and refrained from using the term "selection" in the revised version.

AuthorsOpenReview ↗

Response to Reviewer oQzQ (2/2)

W3: Missing Attention Mask Details. Are action tokens masked from attending to future latent tokens?

We apply a block attention mask in the DiTs same as FastWAM. The future latent attends to current and future latent. The current latent only attends to current latent. The actions attend to actions and current latent. As a result, both current latent and action tokens don't attend to future latent, so we could safely discard the future latent prediction in the inference. The future latent only serves as an additional supervision during training to shape the intermediate representation of latent DiT for the clean frame. We have included an illustration of attention mask in the revised appendix (Fig. 7).

W4: Incomplete Encoder Control Experiments. How does a 4-frame version of DINOv3 perform?

The 4-frame version of DINOv3 result is also added to the table in the revised version. Since DINOv3 is an image encoder, the 4 frames are encoded independently and the latent are concatenated with proper temporal position embeddings as the actualizer inputs. The success rate of 4-frame DINOv3 is 29.8%, even worse than 1-frame DINOv3 (39.7%). The same holds within the V-JEPA family: encoding the same 4 frames independently with the V-JEPA 2.1 image encoder gives 44.8%, versus 60.0% for the video encoder (Table 6). Frame count is therefore not what drives the gain. The future prediction ability from cross-frame encoding is the key. This supports the importance of using world model encoder.

W5: Questionable "Low-Dimensional" Claim

We agree that “inherently low-dimensional” overstates given the results in Fig. 3. We have modified the phrase by directly stating that an actualizer of DiT-S size suffices in the future prediction in V-JEPA 2.1 latent space.

Thanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.

AuthorsOpenReview ↗

Looking forward to your response

Dear Reviewer oQzQ,

We would like to thank you very much for your insightful review, and we hope that our response addresses your previous concerns regarding our paper. However, as the discussion period is expected to end in the next few days, please feel free to let us know if you have any further comments on our work. We would be willing to address any additional concerns you may have.

Thank you again for spending time on the paper, we really appreciate it!

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Paper Revisions

Dear AC and reviewers,

We appreciate your efforts in reviewing our paper. Based on the suggestions of reviewers, we make the following revisions to enrich our paper:

  1. We corrected a transcription error in the RoboActualizer row of Table 2 and unified the Overall metric calculation to the official instance-weighted mean. Currently, the overall success rate on LIBERO-Plus is calculated as the weighted average of each sub-fields following the official procedure in LIBERO-Plus paper. After double check, we fixed some errors in the trainable parameter numbers of baselines in Table 1 and 2.
  2. We followed the reviewer's suggestion to include additional results on RoboTwin 2.0 C2C and C2R benchmark in Table 3, where our method outperformed all baselines without robot data pretraining with the fewest trainable parameters.
  3. We added more baselines to Table 1 and Table 2 including (OpenVLA-OFT, NORA, RIPT-VLA, FOCA), and we also added a column to reflect whether each method is pretrained on large-scale robot data.
  4. We improved the formulation in Eq. 3 to make it more consistent with the implementation MoT architecture.
  5. We included several additional ablation study in Table 6 in response to reviewers' suggestion, including the 4-frame DINOv3 features and several variants of WAN-VAE encoder.
  6. We added an illustration of the block attention mask in Fig. 7 of the appendix.
  7. We changed the terms and polished the descriptions to clearly distinguish between "world model" and "world model encoder".
  8. We revised several paragraphs to avoid potential confusion caused by the term "selection." The current version more clearly describes our method as implicitly determining task-conditioned futures based on the world model representation, rather than explicitly selecting one from multiple rollouts.
  9. We adjusted some descriptions in Sec. 3.3 to avoid the overstatement of low-dimensional latent space.
  10. The anonymous link for code and checkpoints are as follows: https://anonymous.4open.science/r/roboactualizer_review-62E4/ (github) and https://anonymous-hf.com/a/w8244jpvylhg/ (huggingface) .
  11. We fixed some typos in the manuscript.

We sincerely thank all reviewers for their constructive feedbacks which help to polish our paper. We are glad to take any additional questions during the discussion period.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.