acceptodds
Under review as a conference paper at ICLR 2027

FutureDuet: Decoupling Observation Access from Future Supervision in World Action Models

Abstract

World Action Models (WAMs) augment robot action generation with future visual supervision. Existing WAMs commonly fuse main and wrist observations into one visual stream and train both with the same future-video objective, despite their different visual dynamics. A stable main camera reveals scene-level task evolution, whereas wrist cameras move with the end effector, mixing local interaction changes with viewpoint shifts and self-occlusion. These contrasting predictive demands suggest that the two views may benefit from different future objectives. We introduce FutureDuet, which retains both views for control while allowing each visual stream to receive a different future objective. For the main view, future RGB models task evolution, while interaction masks and robot skeletons focus supervision on task objects and robot motion. For the wrist stream, future latent prediction models short-horizon interaction changes without requiring pixel-level reconstruction. ActionDiT jointly reads the resulting Task State and Interaction State, combining scene-level progress with close-range interaction evidence. All auxiliary prediction modules are training-only, adding no inference overhead. FutureDuet achieves 94.2% clean and 94.1% randomized success on RoboTwin50, and 99.2% average success on LIBERO. The improvements are most pronounced on six RoboTwin50 tasks that require precise interaction, averaging gains of 9.2% and 12.8% over Fast-WAM in clean and randomized settings. Controlled studies further show complementary gains from separating the wrist pathway and designing future supervision separately for the two views.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Vision-Language-Action Models, Robot Manipulation, World Models for Control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Gc5VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FutureDuet, a World Action Model (WAM) framework for robot action generation that separately models task evolution from the main view camera and interaction evolution from the wrist view camera with different objectives. Using ActionDiT to read both states, robot actions are generated. This training is evaluated on the RoboTwin50 and LIBERO benchmarks and demonstrates improvement over unified multi-view modeling with existing WAMs.

Critical strengths
  • The motivation for separating the modeling of the two streams in the context of WAMs makes a lot of sense. This idea can be extended to multiview coordination and more fine-grained modeling of latent states from heterogeneous data streams from real-world robots.
  • The experiment section is pretty thorough, with validation on key research questions and ablations. The design choices are carefully validated in factorial studies. The proposed FutureDuet approach shows reasonable performance compared with baselines such as Fast-WAM.
Critical weaknesses
  • In terms of the significance of the approach, the proposed method is essentially latent fusion, and the unified multi-view WAM design (Figure 1(a)) is an early fusion approach that is well studied in the multimodal learning literature. The adaptation of this idea to WAMs is reasonable, but the actual Duet Interface mechanism is simply concatenation and linear transformation. The training mechanisms for the main view and wrist view, as well as the action prediction using Action DiT seem to be pretty standard for WAMs.
  • The performance of the baseline method Fast-WAM already seems to be saturated on both RoboTwin50 and LIBERO benchmarks (Tables 1 and 2). Given the limited headroom, it is unclear how substantial the improvement is on real-world robot environments.
Questions for authors
  • Currently, the framework designs training objectives for each view using view-specific supervision (e.g., mask and skeleton for the main view, V-JEPA features for the wrist view). Is there a universal approach to identify the best training objectives for arbitrary new views added to the system, e.g., an additional camera, sensors, etc.
  • How does FutureDuet handle the case when any of the views suffers from occlusion?
Reviewer_ro5HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FutureDuet assigns different future-prediction objectives to the main and wrist camera streams: RGB and structured targets for the main view, and latent prediction for wrist views. The two streams jointly support action generation, while auxiliary prediction modules are removed at inference. The method is evaluated on LIBERO and RoboTwin50, with additional studies of supervision targets and observation pathways.

Critical strengths
  1. The wrist-pathway study provides useful controlled evidence. Comparing future targets under the same deployed architecture demonstrates the practical value of target selection without adding inference cost.
  2. The benchmark results are strong, with the RoboTwin50 comparison matching Fast-WAM in data, backbone, and training budget. The factorial study also supports the complementary benefits of structured main-view supervision and wrist prediction.
Critical weaknesses
  1. The specific benefit of assigning different objectives to different views remains unclear. The wrist study establishes that V-JEPA targets work better than VAE targets, but does not distinguish future prediction from transferring pretrained V-JEPA features. Current-feature distillation and latent supervision for both views are missing controls; Appendix A.5.1 acknowledges the latter. These comparisons matter because the paper’s main contribution rests on the view-specific assignment, beyond combining useful supervision signals.
Questions for authors
  1. Does assigning different targets to the two views outperform using V-JEPA latent targets for both, under matched training settings?
  2. With the wrist architecture fixed, does predicting future V-JEPA features outperform distilling current V-JEPA features?
Reviewer_JDtEOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new multi-view World Action Model (WAM), a type of model that commonly augments robot action generation with future visual supervision. Many prior multi-view WAMs concatenate observations from different views (the main view and the wrist view) to form a single visual stream and then apply the same future-video objective across views. This paper argues that, since the main and wrist views have their own advantages and disadvantages, it would be better to handle them separately to take advantage of their complementary information. Therefore, this paper introduces a new WAM, named FutureDuet, which decouples future supervision for the two views during training. This training strategy allows the model to learn the task objective and relevant objects and robot motions from the main visual stream. The wrist visual stream helps the model learn about close-range interaction changes. One advantage of the proposed model is that the auxiliary prediction modules for the different views operate only during training, so they do not increase inference costs. The authors evaluate FutureDuet on the RoboTwin50 and LIBERO datasets and report notable performance improvements on both.

Critical strengths

The proposed approach seems simple because its core idea is to decouple the main and wrist visual streams for future-state prediction, since they provide different but complementary information. However, it appears effective, and its effectiveness is generally well supported by experiments on both the RoboTwin50 and LIBERO datasets.

Critical weaknesses

The main concern of this paper is that similar ideas have been explored in prior work, such as decoupling viewpoints to learn view-invariant and view-dependent representations for world model training [1, 2]. In addition, there are some papers with similar core ideas to this paper, although they are only available on arXiv so far [3, 4]. Therefore, the technical novelty of this paper is somewhat limited.

References:

[1] Pang, Jing-Cheng, et al. "Learning view-invariant world models for visual robotic manipulation." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Seo, Younggyo, et al. "Multi-view masked world models for visual robotic manipulation." International Conference on Machine Learning. PMLR, 2023.

[3] Chen, Wenbo, et al. "MVG-WAM: Multiple View Geometry-Aware World-Action Modeling for Robotic Manipulation." arXiv preprint arXiv:2609.37793 (2026).

[4] Huang, Bingqi, et al. "Selective cross-view consistency for world action models: Held-out viewpoint robustness without test-time camera information." arXiv preprint arXiv:2608.21402 (2026).”

Questions for authors

Which specific aspects of FutureDuet distinguish it from the prior work cited above, particularly in how it decouples future supervision across views?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.