- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
UniW(A)M unifies action-conditioned world modeling and four world-action interfaces through a mixture-of-expert structure with shared experts, capability-specific experts, and attention masks. PAIR selects an interface per action chunk while keeping the backbone frozen.
Critical strengths
This submission has a clear motivation for sharing knowledge between world prediction and action generation while supporting different inference costs. Its evaluation covers simulation and real robots, with strong RoboTwin C2R performance and promising real-world results. In experiments, AC-WM ablations consistently support positive transfer to all four action-generation interfaces.
Critical weaknesses
- The conceptual advance over existing unified models remains limited. Prior work already combines world prediction, action generation, and expert specialization. UniW(A)M offers a useful integration, but the paper does not sufficiently establish a distinct modeling principle or capability beyond these existing directions.
- The broader value of unifying multiple interfaces is not convincingly demonstrated. AC-WM co-training provides evidence of positive transfer to action generation. However, this does not establish why jointly maintaining four action interfaces is preferable to a simpler unified model or a well-chosen single interface.
- PAIR’s contribution to generalizable embodied control remains narrow. Its gains rely on target-task supervision, with validation focused on variations within three manipulation tasks. These results support task-specific adaptation, but leave the broader scaling issue of learned interface/experts unverified (Do more experts delivers better compositional generalization?).
Questions for authors
How does PAIR compare with simple phase-based routing or lightweight policy fine-tuning using identical OOD supervision?
3 comments
Response (2/2): Generalization, Heuristic Routing, and LoRA
W3. Generalizability and expert scalability
We thank the reviewer for raising this important concern regarding generalizability and expert scalability. Our current OOD evaluation focuses on variations within three specific manipulation tasks, where PAIR improves average success from 63.3% to 76.7% through supervised target-domain adaptation while keeping the WAM backbone frozen. These results demonstrate the reuse of pretrained capabilities under task-specific distribution shifts.
Regarding expert scalability, our four atomic experts are designed according to the distinct video–action conditional prediction roles described in Section 3.1 and Appendix A.2, rather than simply increasing the number of experts. Our motivation is to organize complementary prediction capabilities for reuse across interfaces, rather than assuming that adding more experts necessarily improves generalization.
More broadly, we believe stronger cross-task and compositional generalization would benefit from substantially larger and more diverse embodied training data. Our current real-world training dataset is relatively limited in scale, containing only three manipulation tasks:
Table 2. Real-world training dataset.
| Task | Episodes | Frames | Duration (h) |
|---|---|---|---|
| Tabletop Cleaning | 2,426 | 1,528,831 | 14.16 |
| Folding Box | 1,217 | 1,768,220 | 16.37 |
| Folding Clothes | 605 | 3,110,732 | 28.80 |
| Total | 4,248 | 6,407,783 | 59.33 |
Despite this task-focused data scale, UniW(A)M achieves an average success rate of 88.3% across the three real-world tasks, compared with 76.7% for the baseline, benefiting from pretrained initialization and unified world–action learning. We consider achieving strong manipulation performance with 59.33 hours of task-specific real-world data an encouraging result. While broader generalization remains an important direction, our findings highlight the effectiveness and data efficiency of jointly learning reusable world–action capabilities in the evaluated setting.
Q1. Heuristic routing and LoRA comparisons
Following your suggestion, we compare PAIR with phase-based heuristic routing and four interface-specific LoRA baselines on the Folding Clothes OOD task.
The heuristic follows the fixed mapping in Appendix B.3: Reach → Action-Only, Contact → Joint, and Core → IDM. Qwen3-VL-8B identifies the current phase based on robot observations and subtask information. In contrast, PAIR learns context-dependent interface selection for each action chunk without relying on predefined phase-to-interface rules.
We independently fine-tune all four WAM interfaces using the same 2,000 OOD training chunks as PAIR, without dynamic routing. Each LoRA adapter uses rank 16, alpha 32, and Kaiming initialization (init_lora_weights="kaiming"), and is injected into the Attention Q/K/V/O projections and FFN layers. Each adapter is trained for 10K optimization steps. The four adapted interfaces and PAIR are evaluated over 20 OOD trials each under the same evaluation protocol.
Table 3. OOD adaptation on Folding Clothes (success rate, %).
| Method | Before LoRA | After LoRA |
|---|---|---|
| Best Fixed Interface | 65.0 | — |
| Phase-based Heuristic | 45.0 | — |
| Joint-WAM | 65.0 | 65.0 |
| IDM-WAM | 65.0 | 65.0 |
| Action-Centric-WAM | 50.0 | 55.0 |
| Action-Only-WAM | 50.0 | 50.0 |
| PAIR (Ours) | 75.0 | — |
Under the same target-domain supervision, LoRA fine-tuning leaves the success rates of Joint, IDM, and Action-Only unchanged, while improving Action-Centric from 50.0% to 55.0%. PAIR achieves 75.0%, compared with 65.0% for both the best fixed interface and the best LoRA-adapted interface, and 45.0% for heuristic routing.
The heuristic results suggest that phase-level preferences can capture general trends but may not provide optimal interface assignments for individual action chunks. In contrast, PAIR selects interfaces based on the current execution context. Its observed 10-percentage-point advantage over the best LoRA baseline supports adapting interface selection as an alternative to fine-tuning individual action predictors under limited target-domain supervision, while keeping the pretrained backbone frozen in this evaluation.
Follow-up on our responses
Dear Reviewer HSQK,
Thank you again for your thoughtful review. In our responses above, we have clarified the conceptual novelty of UniW(A)M, added progressive co-training ablations showing the value of multiple interfaces, and compared PAIR with phase-based routing and LoRA fine-tuning under identical OOD supervision.
As the discussion period is drawing to a close, we would greatly appreciate your feedback. If our responses have resolved your concerns, we would be sincerely grateful if you could reconsider your score in light of the new results. If any point remains unclear, we would be glad to clarify further.
Best regards,
The Authors
Response (1/2): Novelty and Cross-Interface Learning
W1. Conceptual novelty
We thank the reviewer for encouraging us to further clarify the novelty of UniW(A)M.
UWM couples video and action diffusion with flexible conditional inference, Motus integrates understanding, video generation, and action experts, and Worldscape-MoE explores shared and specialized experts for heterogeneous world modeling. UniW(A)M organizes video–action conditional prediction into reusable atomic capabilities, allowing different interfaces to share these capabilities while preserving their distinct conditional dependencies.
Specifically, the atomic experts are organized around prediction roles rather than assigned independently to individual interfaces. VGM serves video tokens across Joint, IDM, and Action-Only, while IDM serves action tokens in both Joint and IDM. Thus, the interfaces are not merely separate predictors collected within one network: they reuse capability-specific computation, enabling supervision from different interfaces to improve common prediction capabilities.
Crucially, expert reuse does not imply identical conditioning information. Attention masks specify what information each interface can access, whereas CATR determines which capability-aligned experts its token groups can use and learns their mixture with shared computation. Masks preserve interface-specific conditional dependencies, while CATR implements cross-interface capability reuse and specialized computation within a single backbone. CATR therefore operationalizes the proposed capability decomposition.
PAIR further connects this shared learning framework to adaptive deployment. It learns a lightweight router for target-domain adaptation, selecting an interface for each action chunk while keeping the unified backbone and individual action predictors frozen. This enables context-dependent use of the jointly learned capabilities.
Our controlled routing and expert-removal ablations support the functional value of this organization, while the cross-interface learning results discussed in W2 address the benefits of jointly maintaining multiple interfaces. We thank the reviewers for their suggestions, which have helped clarify our core contributions.
W2. Benefits of multiple interfaces
We thank the reviewer for raising this important question. The benefits of jointly maintaining multiple WAM interfaces can be understood from two complementary perspectives: cross-interface knowledge transfer during training and adaptive interface selection during inference.
(1) Cross-interface knowledge transfer. We conduct progressive co-training ablations on Folding Clothes to investigate whether jointly training multiple interfaces provides additional benefits. Each configuration is independently trained for 150K total steps from the same initialization and training dataset. Every reported interface is evaluated on 20 matched test scenarios.
Table 1. Progressive interface co-training on Folding Clothes (success rate, %).
As additional interfaces are incorporated into training, Joint improves from 60.0% to 90.0%, IDM from 60.0% to 90.0%, and Action-Centric from 60.0% to 85.0%. These improvements demonstrate the additional benefits of multi-interface co-training under a fixed total training budget. Importantly, incorporating new interfaces improves the performance of existing interfaces, providing evidence of positive cross-interface knowledge transfer beyond AC-WM co-training alone.
(2) Adaptive interface selection during inference. Jointly maintaining multiple WAM interfaces also naturally enables our proposed PAIR mechanism, which adaptively selects the most suitable interface for each action chunk without requiring separately deployed models. By training only a lightweight router while keeping the unified backbone frozen, PAIR improves average OOD success from 63.3% to 76.7% across three manipulation tasks under target-domain adaptation. This demonstrates the practical benefit of maintaining complementary interfaces that can be adaptively utilized according to the current execution context.
Together, these results support the value of our multi-interface design in both training and inference: joint training promotes knowledge transfer across interfaces, while PAIR enables adaptive utilization of their complementary capabilities.