- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes FutureDuet, a World Action Model (WAM) framework for robot action generation that separately models task evolution from the main view camera and interaction evolution from the wrist view camera with different objectives. Using ActionDiT to read both states, robot actions are generated. This training is evaluated on the RoboTwin50 and LIBERO benchmarks and demonstrates improvement over unified multi-view modeling with existing WAMs.
Critical strengths
- The motivation for separating the modeling of the two streams in the context of WAMs makes a lot of sense. This idea can be extended to multiview coordination and more fine-grained modeling of latent states from heterogeneous data streams from real-world robots.
- The experiment section is pretty thorough, with validation on key research questions and ablations. The design choices are carefully validated in factorial studies. The proposed FutureDuet approach shows reasonable performance compared with baselines such as Fast-WAM.
Critical weaknesses
- In terms of the significance of the approach, the proposed method is essentially latent fusion, and the unified multi-view WAM design (Figure 1(a)) is an early fusion approach that is well studied in the multimodal learning literature. The adaptation of this idea to WAMs is reasonable, but the actual Duet Interface mechanism is simply concatenation and linear transformation. The training mechanisms for the main view and wrist view, as well as the action prediction using Action DiT seem to be pretty standard for WAMs.
- The performance of the baseline method Fast-WAM already seems to be saturated on both RoboTwin50 and LIBERO benchmarks (Tables 1 and 2). Given the limited headroom, it is unclear how substantial the improvement is on real-world robot environments.
Questions for authors
- Currently, the framework designs training objectives for each view using view-specific supervision (e.g., mask and skeleton for the main view, V-JEPA features for the wrist view). Is there a universal approach to identify the best training objectives for arbitrary new views added to the system, e.g., an additional camera, sensors, etc.
- How does FutureDuet handle the case when any of the views suffers from occlusion?