acceptodds
Under review as a conference paper at ICLR 2027

MoAct: Learning Visual Transition Dynamics for Robot Action Generation

Abstract

Modeling future observations allows vision-language-action (VLA) models to anticipate scene changes and improve action generation. However, extracting latent representations that retain the information needed for accurate prediction and control remains a key challenge. To address this challenge, we introduce MoAct, a framework that uses compact visual-transition tokens to model changes in future visual context and guide action generation. MoAct learns visual-transition dynamics from large-scale video data and connects a visual-transition dynamics expert with an action expert through a dual-stream Mixture-of-Transformers (MoT) architecture. This design allows predicted scene changes to guide policy learning. Extensive experiments in simulation and the real world show that MoAct outperforms existing future-guided methods. Egocentric human videos also help MoAct retain much of its performance with fewer robot demonstrations. For scene adaptation, MoAct requires only additional egocentric human videos from the target scenes, with no new robot demonstrations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Imitation Learning, Vision-Language-Action Model, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fTMCOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to train VLAs to predict future information for better action generation. The authors focus on transitions between current and future observations, which include motion and spatial relationships during manipulation. In particular, they introduce MoAct, a VLA that learns visual-transition dynamics constructed in compact representations using the MoT architecture. They first train a visual-transition representation using transformer blocks by predicting future DINO features from current DINO features with transition tokens and VLM features. They then predict actions by conditioning these transition representations as conditioning input. They adopt two-stage training that connects visual-transition dynamics learning with task-specific action generation using heterogeneous robot and egocentric human videos.

Critical strengths
  1. The proposed approach shows relatively strong performance only with a 1B-parameter model.
  2. The proposed approach is efficient during inference because the DINO encoder and predictor are not required during inference.
Critical weaknesses
  1. This paper lacks conceptual novelty. Recent work such as LAPA [1] already learns compact representations of visual transitions from action-free videos and leverages them for downstream action learning. Moreover, DUST [2] adopts a dual-stream, MoT-style architecture that jointly models compact future visual representations and future actions. While MoAct differs in how the transition representations are constructed and trained, it remains unclear what fundamentally new capability or insight this formulation provides beyond these closely related approaches.

  2. For the methodogically, this paper lacks an ablation study of core components of the proposed method. Specifically, it does not evaluate the following ablations:

  • Sensitivity analysis of transition tokens by varying the number of them
  • Effect of spatial gating by comparison to a simpler fusion mechanism, and thus it is unclear whether explicitly modeling where the scene changes is useful
  • Effect of layer-wise modulation by removing the FiLM-style modulation, and thus it is difficult to claim the benefit of modeling how the scene changes.
  • Effect of MoT architecture by comparison to a single-stream architecture.
  • Effect of stop-gradient.
  • Effect of two-stage training by comparing with training the full model jointly from scratch.
  1. It is unclear whether the performance of the 1B model comes from the proposed method, because it uses Qwen3.5 as the VLM backbone, but baselines use relatively weaker VLM backbones (e.g., Wog [3] uses Qwen3 as the VLM backbone).

Because the method is relatively complex, the contribution of each component should be carefully identified. Without such ablations, it is difficult to know which parts of the design choices are actually necessary, particularly given the limited conceptual novelty from prior works.

[1] Ye et al., Latent Action Pretraining from Videos, ICLR 2025
[2] Won et al., Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model, ICML 2026
[3] Su et al., World guidance: World modeling in condition space for action generation, ICML 2026

Questions for authors

Please answer the weakness part.

10 comments
AuthorsOpenReview ↗

Response to Reviewer fTMC (Part 1/8)

Thank you for recognizing the strong performance of MoAct with only a 1B-parameter model and its efficiency at inference, where neither the DINOv3 encoder nor the predictor is required. We appreciate the time you spent on our paper.

Before the point-by-point responses, we would like to correct three statements in the review that are central to its assessment:

  • The backbone of WoG is not Qwen3. WoG [3] builds on the Prismatic VLM used by OpenVLA (Section 3.2 of WoG), a 7B model, as listed in Table 1; Qwen3 appears in WoG only as a citation in its related work. More importantly, our paper already isolates the backbone: MoAct (action-only) uses the same Qwen3.5-0.8B initialization and reaches only 51.2% on Google Robot, below π0\pi_0 and π0\pi_0-FAST (Response 8).
  • The paper does contain ablations under a fixed backbone. Section 4.3 compares full MoAct with an action-only variant that shares the same backbone initialization, downstream data, and evaluation protocol (71.7% vs. 51.2% on Google Robot, 65.6% vs. 44.8% on WidowX), and with a variant without human videos in pretraining (68.1% and 62.5%).
  • The stop-gradient is a well-established practice in VLA training. It follows knowledge insulation [4], proposed by the authors of the well-known π\pi series of VLA models, which has been widely adopted in subsequent work and is cited in Section 3.1 of our paper (Response 6).

Beyond these corrections, we have added component-level analyses of the designs you list (Appendix D of the revision):

  • Spatial gating and layer-wise modulation, analyzed on changing and static regions of the successor-prediction target (Responses 3 and 4);
  • The MoT interface, tested by removing or freezing the dynamics keys and values at evaluation time on the released checkpoints (Response 5).

The component-level evidence is summarized in Response 9.

Anonymous resources:

We address your comments point by point below.


References

[3] Y. Su, S. Chen, H. Shi, M. Liu, Z. Zhang, N. Huang, et al. World Guidance: World Modeling in Condition Space for Action Generation. In ICML, 2026. [4] D. Driess, J. T. Springenberg, B. Ichter, L. Yu, A. Li-Bell, K. Pertsch, et al. Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better. In NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer fTMC (Part 2/8)

Response 1 (1/2)

Reviewer's Comment: This paper lacks conceptual novelty. Recent work such as LAPA [1] already learns compact representations of visual transitions from action-free videos and leverages them for downstream action learning. Moreover, DUST [2] adopts a dual-stream, MoT-style architecture that jointly models compact future visual representations and future actions. While MoAct differs in how the transition representations are constructed and trained, it remains unclear what fundamentally new capability or insight this formulation provides beyond these closely related approaches.

Response: We respectfully disagree. The core idea of MoAct is that imagining only the part of the future that changes is sufficient to improve action generation. This makes MoAct highly efficient, and because learning such changes requires no action labels, it allows low-cost egocentric human videos to improve downstream manipulation (Section 4.2). We believe this offers a promising route to exploiting egocentric human data and to alleviating the cost of teleoperated robot data. As their own descriptions show, LAPA and DUST differ from MoAct in what the compact representation encodes, how it is learned, and whether the deployed policy uses it.

Relative to LAPA [1]: MoAct's transition tokens are a continuous, learned condition that the deployed policy reads at every step, rather than discrete pseudo-labels that are discarded.

  • LAPA's latent actions are discrete codes produced by a separate quantization model. LAPA first trains a VQ-VAE whose encoder "takes the current frame xtx_t and the future frame xt+Hx_{t+H} ... and outputs the latent action ztz_t" (LAPA, Section 3.1), and the difference of the two frame embeddings is discretized "by finding the closest embedding from the codebook" (LAPA, Appendix A). This frozen encoder then serves "as an inverse dynamics model to label all frames" for pretraining (LAPA, Section 3.2). In MoAct, there is no codebook and no separate labeling model: the continuous transition tokens are produced by the VLA itself from the current observation and instruction, and are learned end to end through the successor-feature prediction loss.
  • LAPA's latent actions are removed before deployment. LAPA states that "VLAs that are pretrained to predict latent actions are not directly executable on real-world robots", and therefore "we discard the latent action head ... and replace it with a new action head to generate ground truth actions" (LAPA, Section 3.3). The latent actions are thus a pretraining target only. In MoAct, the transition tokens are recomputed at every control step and condition the action expert through layer-wise keys and values, and the deployed policy depends on them: freezing them at their first-step values reduces success from 71.7% to 20.6% on Google Robot and from 65.6% to 41.7% on WidowX, and removing them reduces success to 26.4% and 15.6% (Appendix D.2).

References

[1] S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, et al. Latent Action Pretraining from Videos. In ICLR, 2025. [2] J. Won, K. Lee, H. Jang, D. Kim, and J. Shin. Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model. In ICML, 2026.

AuthorsOpenReview ↗

Response to Reviewer fTMC (Part 3/8)

Response 1 (2/2)

Relative to DUST [2]: MoAct conditions actions on the predicted change, while DUST generates the future state as a second diffusion modality.

  • DUST's target is the future state itself. DUST defines its world-modeling target as "the representation of the future observation ot+kvo^v_{t+k} produced by the vision encoder", the observation "obtained after executing the action chunk" (DUST, Section 3), namely "the future image embedding derived from the SIGLIP-2 representations" (DUST, Section 5). In MoAct, the predictor receives the preceding grid Fi−1\mathbf{F}_{i-1} as an explicit reference, so each token only needs to explain the change to Fi\mathbf{F}_i rather than re-encode the future scene.
  • DUST generates future embeddings at inference; MoAct generates none. In DUST, the diffusion model "takes the triplet (ots,Atτ,o~t+kτ)(o^s_t, A^\tau_t, \tilde{o}^\tau_{t+k}) as input, ... composed of the robot state, noised actions, and noised future observation embeddings" (DUST, Section 4.1), and "during inference, we jointly sample actions and vision in parallel" (DUST, Section 4.3). In MoAct, the dynamics stream is not a diffusion modality: it is computed once per control step without noise and reused across all flow-integration steps, and the DINOv3 encoder and predictor are discarded at inference (Appendices A.3–A.4).
  • The two dual-stream designs exchange information differently. In DUST, the action and vision streams "are concatenated ... during the shared cross-modal attention layer", so that "generated actions constrain plausible future states, and predicted states guide action generation" (DUST, Sections 4.1 and 4.3). In MoAct, information flows one way: the action queries attend to the dynamics keys and values under stop-gradient, so the action loss does not propagate back into the dynamics stream (Section 3.1).

The tokens carry change-specific information, as the formulation intends. On 1,024 samples each from Fractal and Bridge, the tokens reduce the successor-prediction error on changing regions by 39% and 37% relative to copying the preceding features, while matching the copy baseline on static regions (Appendix D.1). The compact representation thus carries the information about where and how the scene changes, and static content is propagated from the reference.


References

[2] J. Won, K. Lee, H. Jang, D. Kim, and J. Shin. Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model. In ICML, 2026.

Reviewer_KBNdOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MoAct learns compact visual-transition representations from robot and egocentric human videos and uses them to guide continuous action generation. Eight transition tokens are supervised by predicting successor DINOv3 features from preceding observation features, using spatial gating and feature modulation. A dual-stream Mixture-of-Transformers then supplies dynamics information to a flow-matching action expert. The two-stage training procedure combines video-based transition pretraining with downstream transition and action learning; human videos contribute visual supervision without requiring robot action labels. The paper evaluates the approach on two SimplerEnv robot embodiments and four real-world manipulation tasks, and studies both reduced robot-demonstration requirements and adaptation using newly collected human videos.

Critical strengths
  1. The representation and its connection to control are well motivated. I particularly like the use of preceding observation features as a reference for predicting the next visual context. This gives the transition tokens a concrete learning objective and encourages them to describe scene evolution. The spatial gates and transition-conditioned modulation provide a clear implementation of this idea, while the MoT interface lets the action expert use predictive information together with the current observation and robot state. The design also takes deployment seriously: the dynamics keys and values are reused during flow integration, and neither DINOv3 nor the feature predictor is needed for action inference. Learning from detailed visual targets while passing only eight transition tokens to the policy is an appealing design choice.

  2. The empirical results make a convincing case that the overall approach is useful. MoAct reports the best aggregate success rates among the compared methods on both Google Robot and WidowX, at 71.7% and 65.6%, with a model of approximately 1B parameters. The gains over the action-only variant are substantial on both embodiments. I find the physical-robot experiments especially valuable: average success reaches 74.0% versus 65.0% for the pi0.5 comparison using the same number of downstream demonstrations. More compelling still, adding 50 human videos to 50 robot demonstrations raises average success from 58.0% to 68.5%. The scene-adaptation experiment provides a concrete example of improving an existing policy using human videos as the only newly collected data, while retaining the original robot demonstrations. Together, these results support a practical route for making use of video data in robot learning.

I support acceptance. The paper connects a clear representation-learning idea to measurable benefits in control and data collection, and the combination of simulation, physical manipulation, and human-video adaptation gives the contribution substance.

Critical weaknesses

The main limitation is that the ablations establish the value of the complete learning pipeline more clearly than the contribution of each component. As Appendix B.6 explains, the action-only variant removes both transition pretraining and transition supervision during fine-tuning. This leaves room for a more precise analysis of the roles of spatial gating, feature modulation, and the MoT interface. The scene-adaptation evidence is also limited to three variations of package storage. These points bound the conclusions, but I do not consider them sufficient reasons to reject the paper given the demonstrated system-level benefits.

Questions for authors
  1. Were the simulation baseline results reproduced under the same action-execution and evaluation conventions as MoAct, or taken from the cited works? A short clarification of which settings were shared would help interpret the comparison with the strongest baselines.

  2. In the scene-adaptation experiment, is the 66.7% reference the original checkpoint, or a checkpoint continued for the same number of updates using only the existing robot demonstrations? If the latter control is available, it would help establish how much of the improvement comes specifically from the new human videos.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KBNd (Part 1/2)

Thank you for recognizing the clear motivation of the transition representation, the deployment-aware design of MoAct, and the practical value of our real-world and human-video results. We are grateful for your support for acceptance, and we sincerely appreciate your time, your careful reading of our paper, and your constructive suggestions, which have helped us improve it.

Following your suggestions, we have made the following additions and clarifications:

  • Component-level analysis. We added analyses of spatial gating, feature modulation, and the MoT interface (Response 1).
  • Scope of scene adaptation. We agree with the scope you point out and have noted in the Limitations section that we will evaluate scene adaptation on a broader range of scenes (Response 2).
  • Baseline settings and the scene-adaptation setup. We clarify the source and shared settings of the baseline results (Response 3), and the reference checkpoint and purpose of the scene-adaptation experiment (Response 4).

Anonymous resources:

We address your comments point by point below.


Response 1

Reviewer's Comment: The main limitation is that the ablations establish the value of the complete learning pipeline more clearly than the contribution of each component. As Appendix B.6 explains, the action-only variant removes both transition pretraining and transition supervision during fine-tuning. This leaves room for a more precise analysis of the roles of spatial gating, feature modulation, and the MoT interface.

Response: Thank you for this very helpful suggestion. We agree that the main ablation speaks to the pipeline as a whole, and following your comment, we have added component-level analyses of the three designs you mention (Appendix D).

  • Spatial gating and feature modulation. On the Stage 1 checkpoint behind the reported models, we feed one pathway with the transition tokens of another sample while keeping the other intact, or disable it, and measure successor-prediction error on changing and static regions (1,024 samples each from Fractal and Bridge):
Successor-prediction MSEFractal changingFractal staticBridge changingBridge static
Full predictor0.1290.0220.1200.019
Gate from another sample0.1400.0230.1270.020
Modulation from another sample0.1760.0290.1660.030
Gate fixed to one0.1520.0400.1380.038
Modulation removed0.3850.1900.3520.177

Relative changes in this response are computed from unrounded values.

  • The two designs play complementary roles, in line with the intuition you describe. Feature modulation carries most of the sample-specific transition information: replacing only its input accounts for about 80% of the changing-region error increase caused by shuffling the entire token, versus 19%/14% for the gate (Fractal/Bridge). Spatial gating, in turn, preserves static content: fixing the gate to one nearly doubles the static-region error. The gate thus determines where the preceding features are preserved, and the modulated layers encode the change.
  • The MoT interface carries step-wise transition information to the action expert. On the released checkpoints, removing the dynamics keys and values from the joint attention at evaluation time reduces success from 71.7% to 26.4% on Google Robot and from 65.6% to 15.6% on WidowX, and freezing them at their first-step values reduces success to 20.6% and 41.7%. This supports your reading that the action expert makes use of the predictive information together with the current observation and robot state.
AuthorsOpenReview ↗

Response to Reviewer KBNd (Part 2/2)

Response 2

Reviewer's Comment: The scene-adaptation evidence is also limited to three variations of package storage. These points bound the conclusions, but I do not consider them sufficient reasons to reject the paper given the demonstrated system-level benefits.

Response: Thank you for this fair assessment and for your encouraging view of the system-level benefits.

  • We agree that the scene-adaptation evidence is limited to three variations of package storage. We view this study as a demonstrated case of adapting a deployed policy with human videos, and we have noted in the Limitations section of the revision that we will further evaluate scene adaptation on a broader range of scenes.
  • Within this scope, the three target scenes change the manipulated objects, the container, or both (towels with a box, packages with a basket, and towels with a basket). Videos of the adapted policy in all three scenes are available on our anonymous project page (https://moact-anon.github.io/moact-page/).

Response 3

Reviewer's Comment: Were the simulation baseline results reproduced under the same action-execution and evaluation conventions as MoAct, or taken from the cited works? A short clarification of which settings were shared would help interpret the comparison with the strongest baselines.

Response: Thank you for this helpful question; we agree that this clarification makes the comparison easier to interpret.

  • The baseline results are taken from the cited works, with corrected averages. All baseline numbers in Tables 1 and 2 come from Tables 1 and 2 of WoG [1]. While compiling them, we found that several averages reported there do not match the listed per-task values; for example, the average WidowX grasp rates of π0, π0-FAST, OpenVLA, and GR00T N1 (40.1%, 48.3%, 7.8%, 49.5%) should be 53.1%, 64.6%, 14.5%, and 62.5%, and the Google Robot overall score of OpenVLA (33.8%) should be 36.3%. We therefore recomputed all averages from the listed per-task values, and the source of the baseline results is stated in the Baselines paragraph of Section 4. These corrections do not affect the comparison with MoAct.
  • The evaluation settings are consistent across methods. All methods are evaluated on the same SimplerEnv task suites, with the same Visual Matching and Variant Aggregation splits for Google Robot, the same four WidowX tasks, and the success criteria defined by the environments.

Response 4

Reviewer's Comment: In the scene-adaptation experiment, is the 66.7% reference the original checkpoint, or a checkpoint continued for the same number of updates using only the existing robot demonstrations? If the latter control is available, it would help establish how much of the improvement comes specifically from the new human videos.

Response: Thank you for this insightful question.

  • The 66.7% reference is the original checkpoint before adaptation, i.e., the MoAct policy trained on 100 robot demonstrations and evaluated directly in the three target scenes. The adapted model continues training from this checkpoint with the same robot demonstrations, adding only the target-scene human videos (Appendix B.3). We have stated this in Section 4.2.
  • The experiment is designed to show a low-cost route to scene adaptation. Adapting a deployed policy to new scenes usually requires collecting new robot demonstrations through teleoperation. In our setting, the only new data are 50 egocentric human videos recorded with a single GoPro camera, without a robot or teleoperation, and the adapted policy improves from 66.7% to 76.0% in the target scenes without any new robot demonstrations.
  • We agree that the control you describe would further isolate the contribution of the human videos. We thank you for this suggestion and will include it in our future evaluation of scene adaptation.

We appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.


References

[1] Y. Su, S. Chen, H. Shi, M. Liu, Z. Zhang, N. Huang, et al. World Guidance: World Modeling in Condition Space for Action Generation. In ICML, 2026.

Reviewer_tnwxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoAct learns compact visual-transition tokens by predicting successor DINO features from preceding features and a transition token. A spatial gate and feature modulation implement the transition predictor, and an action expert conditions on the resulting dynamics stream. The training pipeline uses robot and egocentric human videos before downstream action learning. Evaluation covers SimplerEnv and real-robot manipulation, including limited-demonstration and human-video settings.

Critical strengths
  1. Predicting changes relative to preceding visual features is a reasonable inductive bias: it may reduce the need to encode unchanged appearance in a compact token. The implementation is lightweight, and the full system obtains competitive results on two robot embodiments in SimplerEnv and on physical manipulation tasks.
  2. The human-video comparisons provide meaningful evidence of practical utility. Removing human videos reduces the reported SimplerEnv success from 71.7%/65.6% to 68.1%/62.5%. The real-robot limited-demonstration experiment also benefits from added human videos. These results support the usefulness of the additional video data within the proposed pipeline.
Critical weaknesses
  1. The distinguishing transition mechanism is not isolated. FLARE [R1] already couples future-feature learning to action generation and human-video training; World Guidance [R6] learns compact, action-relevant future conditions and is already included in the paper's comparisons. MoAct's specific increment is therefore the preceding-feature-conditioned transition representation with spatial gating and modulation, not latent prediction or human-video transfer alone. Yet qualitative gate visualizations and task-level successes do not establish the separate benefit of gating, modulation, or change-focused tokens. Since the predictor also receives the preceding feature grid, low whole-grid prediction error could partly reflect static content.
  2. The principal ablation tests a training pipeline rather than the claimed representation mechanism. The action-only variant skips Stage 1 and initializes the VLM differently (Appendix B.6); its 51.2%/44.8% versus 71.7%/65.6% gap cannot be assigned specifically to visual-transition modeling. The authors correctly describe this as a whole-pipeline comparison, but that qualification leaves the novel mechanism without a matched test. Likewise, adding human videos tests the value of additional data, not whether temporal dynamics, rather than scene exposure or additional optimization, explains the transfer. These gaps are central because the broad predictive-learning motivation is already well established.
  3. The release lacks complete workflows for real-world deployment, mixed robot–human fine-tuning, scene adaptation, and RTC execution. The evaluation pipeline also contains confirmed execution-blocking errors. Without trained checkpoints, episode-level results, and dataset inventories, several central experimental claims cannot be independently verified.
  4. The evaluator treats opening and closing drawers as separate tasks, increasing Drawer’s weight from one-third to one-half relative to the stated protocol. Released defaults also differ from the reported precision and global batch size: FP32 versus BF16 and 256 versus 1,024. Table 1 additionally reports OpenVLA’s overall score as 33.8%, although its listed VM/VA averages yield 36.25%. These discrepancies require clarification and corrected reproducibility materials.

[1] Zheng, Ruijie, et al. "Flare: Robot learning with implicit world modeling." arXiv preprint arXiv:2505.15659 (2025).

[2] Su, Yue, et al. "World guidance: World modeling in condition space for action generation." arXiv preprint arXiv:2602.22010 (2026).

Questions for authors
  1. With identical Stage 1 data, initialization, and budget, does a generic future-feature predictor perform worse than the proposed transition predictor? A focused gate/modulation control would help establish which added design is necessary.
  2. Does the successor predictor use the transition token beyond what the preceding features alone provide? A current-feature-only or token-shuffling diagnostic, preferably evaluated on changing regions as well as the whole grid, would directly address this question.
  3. Is there an equal-budget control that preserves the added human images but removes their valid temporal correspondence? Such a comparison could distinguish temporal transfer from scene familiarity. If unavailable, the transfer claim should remain at the level of beneficial additional video data. Please explicitly address the reproducibility gaps and inconsistencies between the manuscript, code, and reported results with supporting evidence; otherwise, I will consider lowering my score.
9 comments
AuthorsOpenReview ↗

Response to Reviewer tnwx (Part 1/7)

Thank you for recognizing the soundness of our core design, the efficiency and competitiveness of the overall system, and the practical value of learning from human videos. We sincerely appreciate your time and effort in reviewing our paper, as well as your detailed and constructive feedback.

We have carefully addressed each of your concerns, and the new experiments further support the central claim of the paper. We thank you again for your contribution to improving the quality of this work.

Your comments raised two central concerns:

  • Isolating the transition mechanism. We clarified that the existing ablation already measures the contribution of visual-transition learning under the same backbone initialization (Response 4), and added two types of evidence:

    • token diagnostics evaluated on the whole grid and separately on changing and static regions (Responses 3, 14);
    • evaluation-time interventions on the released checkpoints (Response 2).

    Together, these results indicate that the learned tokens encode change-specific information beyond the preceding features, and that the policy relies on this information at every control step.

  • Reproducibility. We have updated the released materials with the trained checkpoints and verified that the released pipeline runs end to end from a clean installation. The reported inconsistencies concern documentation and result-aggregation details, and none of them affects the conclusions of the paper (Responses 6–11, summarized in Response 16).

Anonymous resources:

We address your comments point by point below.

AuthorsOpenReview ↗

Response to Reviewer tnwx (Part 2/7)

Response 1

Reviewer's Comment: The distinguishing transition mechanism is not isolated. FLARE [R1] already couples future-feature learning to action generation and human-video training; World Guidance [R6] learns compact, action-relevant future conditions and is already included in the paper's comparisons. MoAct's specific increment is therefore the preceding-feature-conditioned transition representation with spatial gating and modulation, not latent prediction or human-video transfer alone.

Response: Thank you for helping us delineate our contribution more precisely.

  • This increment is exactly the representational contribution the paper claims. MoAct is defined by its representation of scene evolution "as transitions between adjacent observations, rather than as independently predicted future states" (Section 3.1). We do not claim latent prediction per se as novel, and human videos serve as training data within our recipe; what distinguishes MoAct's representation is its transition formulation.
  • Rather than reconstructing the complete future state, MoAct explicitly models the future as compact visual transitions. FLARE [1] aligns policy features with future-observation features, and World Guidance [2] learns future conditions through action supervision; in both, the representation targets the future state itself and thus carries information about what the future scene looks like. MoAct instead provides the predictor with the preceding feature grid as an explicit reference, so each of its eight transition tokens only needs to encode how the scene changes. The future is thus modeled explicitly, but in a more compact form than the full future state.
  • This formulation also makes future modeling efficient at inference. The transition tokens enter the action expert as layer-wise keys and values recomputed at every control step, no future features are decoded, and neither DINOv3 nor the feature predictor is needed; with about 1B parameters, MoAct generates a 32-step action chunk in 54.8 ms (Appendix A.4).
  • The learned tokens are measurably change-specific. On changing regions, they reduce successor-prediction error by 39%/37% relative to a no-change baseline that uses the preceding features alone, while giving no advantage on static regions (Responses 3 and 14). The policy relies on this information at every step: freezing the dynamics conditions at their first-step values reduces Google Robot success from 71.7% to 20.6% (Response 2).
  • We have made this positioning explicit in the revision. The related-work section now states the distinction from FLARE and World Guidance, and the new analyses have been added to Appendix D.

Response 2

Reviewer's Comment: Yet qualitative gate visualizations and task-level successes do not establish the separate benefit of gating, modulation, or change-focused tokens.

Response: Thank you for pointing this out.

  • The visualization in Figure 3 was intended as an illustration, not as evidence of separate benefits. It shows how a token transforms the preceding features into the successor features on one example; the quantitative evidence in the paper is the system-level comparison.
  • We now provide quantitative evidence at two levels: what the tokens encode, and whether the policy uses them.
  • The tokens are change-focused. On changing regions, the true tokens reduce successor-prediction error by 39%/37% relative to the no-change baseline and by 31%/32% relative to shuffled tokens (Fractal/Bridge), while the true tokens and the no-change baseline perform on par on static regions (Response 3).
  • The policy depends on dynamics conditions that are updated at every step. On the released checkpoints, freezing the dynamics conditions at their first-step values reduces Google Robot overall success from 71.7% to 20.6% and WidowX success from 65.6% to 41.7%; removing the dynamics keys and values reduces them to 26.4% and 15.6% (Appendix D.2).

References

[1] R. Zheng, J. Wang, S. Reed, J. Bjorck, Y. Fang, F. Hu, et al. FLARE: Robot Learning with Implicit World Modeling. In CoRL, 2025. [2] Y. Su, S. Chen, H. Shi, M. Liu, Z. Zhang, N. Huang, et al. World Guidance: World Modeling in Condition Space for Action Generation. In ICML, 2026.

AuthorsOpenReview ↗

Response to Reviewer tnwx (Part 3/7)

Response 3

Reviewer's Comment: Since the predictor also receives the preceding feature grid, low whole-grid prediction error could partly reflect static content.

Response: Thank you for this important point.

  • The paper does not use whole-grid prediction error as evidence for the mechanism. No whole-grid error is reported, and Figure 3 is a qualitative illustration. Since whole-grid error would indeed be dominated by static content, we evaluate the predictor region by region, and report the whole-grid error alongside for reference.
  • Region-wise evaluation separates static content from change. For each transition, we measure the per-patch change between the preceding and successor DINOv3 features; changing regions are the 25% of patches with the largest change, and static regions are the 50% with the smallest change. Using the Stage 1 checkpoint from which the reported models were fine-tuned, we evaluate 1,024 Fractal and 1,024 Bridge samples:
Successor-prediction MSEFractal wholeFractal changingFractal staticBridge wholeBridge changingBridge static
True tokens0.0610.1290.0220.0560.1200.019
Shuffled tokens (from other samples)0.0850.1880.0350.0810.1770.033
No-change baseline (preceding features as prediction)0.0830.2130.0220.0760.1900.019

Relative changes in this response are computed from unrounded values.

  • On static regions, the tokens provide no advantage, as expected. The true tokens match the no-change baseline (0.022 vs. 0.022; 0.019 vs. 0.019).
  • On changing regions, the tokens reduce the error substantially. The error is 39%/37% lower than the no-change baseline and 31%/32% lower than with shuffled tokens (Fractal/Bridge).
  • The whole-grid error understates this effect, confirming your concern about static content. On the whole grid, the reduction relative to the no-change baseline shrinks to 27%/26%, because static patches dilute the change; this is why we report the regions separately.
  • The predictive benefit therefore comes from change-specific information rather than static content. We have added this table and the region definition to Appendix D.1.

Response 4

Reviewer's Comment: The principal ablation tests a training pipeline rather than the claimed representation mechanism. The action-only variant skips Stage 1 and initializes the VLM differently (Appendix B.6); its 51.2%/44.8% versus 71.7%/65.6% gap cannot be assigned specifically to visual-transition modeling. The authors correctly describe this as a whole-pipeline comparison, but that qualification leaves the novel mechanism without a matched test.

Response: Thank you for this comment. We would like to clarify a point about this comparison that may not have been sufficiently visible in the main text.

  • The action-only variant does not use a different VLM initialization. As stated in Appendix B.6, it initializes the VLM backbone from the same original Qwen3.5-0.8B checkpoint used to initialize full MoAct before Stage 1. It differs only in not inheriting the parameters updated during Stage 1, which is precisely the effect being measured. We have now also stated this explicitly in Section 4.3.
  • Skipping Stage 1 is not a separate factor, because Stage 1 consists solely of visual-transition learning. Its only objective is the transition loss Ltrans\mathcal{L}_{\mathrm{trans}} (Section 3.2 and Appendix B.2); it involves no action supervision, captioning, or other auxiliary objective. Removing Stage 1 therefore removes visual-transition learning together with the data it is trained on; no other objective is involved.
  • The comparison thus measures the contribution of visual-transition learning as a pretraining objective. The two models share the same backbone initialization, downstream demonstrations, and evaluation protocol; the full model additionally learns visual transitions, in Stage 1 and through the transition objective in Stage 2. The 20.5- and 20.8-point gains on Google Robot and WidowX therefore reflect what visual-transition learning contributes, and Responses 3, 13, and 14 further examine what the learned transitions encode.
  • What the learned transitions encode is further examined in Responses 3 and 14. The tokens carry sample-specific information concentrated on changing regions, beyond what the preceding features provide.

Comments on the paper

1 comment
Area_Chair_Y7xXOpenReview ↗

Discussion with the authors

Dear reviewers,

the authors have provided responses for your reviews. Can you please let them know if your comments have been addressed?

Thank you, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.